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DEVELOPING  TRAINING  EXEMPLARS  FOR  THE  REQUISITE  COMPONENTS  OF 
VISUAL  THREAT  DETECTION 

EXECUTIVE  SUMMARY _ 


Research  Requirement: 

Visual  threat  detection  continues  to  be  a  necessary  task  carried  out  by  Soldiers  in  the 
operational  environment  on  the  majority  of  (if  not  every)  mission.  In  this  research,  the  primary 
processes  of  visual  threat  detection  were  explored  to  improve  understanding  and  provide  a 
substantive  model  upon  which  further  research  could  be  conducted.  As  Soldiers  are  tasked  to 
operate  within  and  around  local  populations,  it  is  imperative  that  they  have  a  solid  understanding 
of  what  cues  represent  likely  threats  and  deserve  the  appropriate  attention  and  response. 

Procedure: 

Stimuli  were  refined  from  previous  research  and  used  in  a  final  experiment.  The  final 
experiment  allowed  for  a  test  of  the  threat  detection  loop  which  is  the  model  of  expert  threat 
detection  developed  in  the  first  phases  of  this  research.  Soldiers  participated  in  several 
computer-based  exercises  and  then  took  part  in  semi-structured  interviews  where  they  provided 
feedback  on  their  experiences  involving  visual  threat  detection.  Those  results  were  incorporated 
with  the  previous  findings  and  were  used  in  the  development  of  a  training  exemplar.  The 
exemplar  was  refined  based  on  a  formative  evaluation  in  which  Soldiers  and  trainers  provided 
feedback  on  usability  and  content. 

Findings: 

The  results  revealed  that  the  threat  detection  loop  is  a  viable  conceptualization  of 
experience-based  threat  detection.  As  Soldiers  gain  experience  with  the  primary  components  of 
threat  detection,  they  develop  expertise  and  are  better  able  to  discern  what  constitutes  a  valid 
threat  in  operational  settings.  Both  quantitative  and  qualitative  results  suggest  that  experience,  as 
described  by  the  model,  does  positively  impact  the  ability  to  visually  search  for  and  make  a 
determination  about  threat  cues  and  overall  threatening  situations.  Soldiers  and 
instructors/trainers  provided  constructive  feedback  on  the  training  exemplar;  they,  likewise, 
noted  that  the  exemplar  would  be  useful  for  newer,  less  experienced  Soldiers  as  well  as  for 
combat  veterans  who  might  need  re-training  on  threat  detection  skills  prior  to  deployment. 

Utilization  and  Dissemination  of  Findings: 

The  model  developed  for  this  research  can  be  used  to  guide  further  research  on  visual 
threat  detection.  The  materials  developed  to  test  the  model  can  also  be  used  to  explore  future 
questions  to  improve  understanding  of  visual  threat  detection.  The  research  based  training 
exemplar  can  be  distributed  to  operational  units  to  bolster  existing  training  or  stand  alone  as  an 
example  demonstrating  the  primary  components  of  visual  threat  detection.  The  exemplar  was 
transitioned  to  a  deployed  Brigade  Combat  Team  (BCT)  to  enhance  their  training  in  theater  as 
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well  as  to  a  training  support  battalion  to  augment  extant  training  that  battalion  offered  as  part  of 
their  pre-mobilization  training  regimen. 
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DEVELOPING  TRAINING  EXEMPLARS  FOR  THE  REQUISITE  COMPONENTS  OF 

VISUAL  THREAT  DETECTION 

Introduction 

Threat  detection  is  an  important  skill  that  United  States  Anny  Soldiers  must  utilize  when 
working  in  the  Operational  Environment  (OE).  Recognizing  that  the  ability  to  detect  threats  is 
vital,  the  Army  has  focused  on  developing  technologies  to  assist  Soldiers  in  extending  this 
capability.  While  certain  technologies,  like  unmanned  aerial  vehicles,  can  be  particularly  useful 
for  providing  a  bird’s  eye  view  of  an  area  of  operations,  they  may  not  always  be  available. 
Moreover,  these  technologies  are  intended  to  augment,  not  replace  the  human  ability  to  detect 
threats.  Threat  detection  relies  on  a  number  of  perceptual  skills  such  as  attention  management, 
pattern  matching,  and  change  detection  as  well  as  higher-order  cognition.  Humans  must  reason 
about  the  threats,  incorporating  their  previous  experiences  and  knowledge  about  the  context  to 
assess  whether  a  particular  cue  might  indicate  a  potential  threat. 

A  large  body  of  scientific  research  provides  empirical  evidence  that  can  guide  our 
understanding  of  how  these  skills  should  manifest  in  the  OE  with  Soldiers  of  varying  experience 
levels.  Leveraging  this  research  base,  several  related  research  projects  were  conducted  to  better 
understand  and,  ultimately,  improve  visual  threat  detection  performance  in  the  OE.  By  relying 
on  current  psychological  theories  of  visual  attention  and  expertise,  military  doctrine,  and 
Soldiers’  threat  detection  experiences  during  recent  deployments,  this  research  aimed  to  achieve 
a  greater  understanding  of  how  Soldiers  detect  threats  in  the  OE.  Culmination  of  that 
information  resulted  in  the  development  of  a  threat  detection  model,  the  Threat  Detection  Loop, 
which  is  discussed  in  the  next  section.  The  objective  was  to  use  this  improved  understanding  of 
the  threat  detection  process  to  create  training  to  enhance  Soldiers’  threat  detection  performance. 
The  goals  of  this  research  were  to: 

•  identify  the  cues,  skills,  and  strategies  required  to  detect  threats  in  the  OE, 

•  differentiate  the  cognitive  skills  employed  by  experienced  versus  novice  Soldiers,  and 

•  create  training  exemplars  that  enhance  the  cognitive  threat  detection  skills  of  Soldiers. 

This  report  presents  a  culminating  experiment  intended  to  tie  together  our  research  in  this 
area  and  provides  details  about  the  training  exemplar  developed  to  enhance  Soldiers’  threat 
detection  skills.  The  purpose  of  the  experiment  was  to  test  the  cognitive  and  perceptual  skills  of 
Soldiers  who  had  varying  levels  of  experience  detecting  threats  in  the  OE.  Soldiers  in  this 
experiment  had  experience  levels  ranging  from  zero  to  four  deployments.  To  test  their  skills, 
Soldiers  completed  a  series  of  computer  exercises  that  examined  their  performance  while  doing 
(a)  prioritized  threat  search,  (b)  dynamic  threat  detection,  and  (c)  change  detection.  This 
research  also  sought  to  identify  any  differences  in  reasoning  about  potential  threats  in  the  context 
of  various  OE  relevant  situations.  Soldiers  participated  in  interviews  and  discussed  the  threat 
cues  and  situational  elements  depicted  in  photographs  of  U.S.  troop  activities  and  terrain  in  Iraq 
and  Afghanistan. 

A  previous  report  documented  the  procedures  and  findings  of  the  first  research  phases 
conducted  (Zimmerman,  Mueller,  Grover,  &  Vowels,  In  Preparation).  The  initial  step  toward 
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understanding  threat  detection  in  the  OE  was  to  review  the  scientific  literature  and  gather 
information  about  the  cognitive  and  perceptual  processes  that  factor  into  threat  detection. 
Analysis  of  this  research  provided  insight  into  the  mechanisms  that  lead  to  threat  detection  and 
guidance  on  how  to  improve  threat  detection.  Three  preliminary  research  phases  followed  the 
literature  review.  The  purpose  of  these  research  phases  was  to  investigate  the  cognitive 
processes  of  Soldiers  during  threat  detection  exercises  and  gather  information  from  experienced 
Soldiers  about  threat  detection  in  the  OE.  Findings  from  these  research  phases  formed  the 
foundation  of  the  current  experiment  and  the  training  exemplar. 

The  research  in  Phase  I  provided  initial  understanding  of  threat  detection  in  the  OE. 
Soldiers  at  Fort  Hood,  TX,  participated  in  interviews  and  completed  questions  (Phase  la)  and 
police  officers  on  patrol  participated  in  ride-along  observations  (Phase  lb).  This  initial  research 
provided  information  about  threat  detection  situations  and  activities  in  the  OE  and  the  case-study 
account  of  patrol  activities  by  police  officers  provided  insight  into  situations  with  high  threat 
likelihood.  The  results  from  this  research  supported  the  findings  highlighted  in  the  literature 
review  about  the  importance  of  threat  search,  change  detection,  and  attention  management  when 
trying  to  detect  threats.  Observations  of  police  officers  with  varying  levels  of  experience 
provided  examples  of  threat  detection  that  occurred  in  their  patrol  environments.  Those 
environments  required  constant  vigilance  to  detect,  assess,  and  react  to  potential  threats.  The 
findings  illustrated  possible  differences  in  threat  detection  processes  of  less  and  more 
experienced  threat  detectors. 

In  Phase  II,  Soldiers  provided  data  by  completing  an  imagery  analysis  exercise, 
questionnaires,  and  in-depth  interviews.  This  allowed  for  a  test  and  refinement  of  research 
materials  and  development  of  a  threat  detection  model.  In  the  imagery  analysis  exercise, 

Soldiers  identified  areas  of  potential  threats  and  annotated  these  areas  with  reasons  why  these 
areas  were  a  concern.  This  analysis  provided  information  about  the  regions  of  interest,  threat 
cues,  and  explanations  about  the  threats.  From  this,  materials  were  created  to  measure  attention 
management,  threat  search  activities,  and  causal  reasoning  about  threat  environments.  In  the 
questionnaires,  Soldiers  from  combat  arms,  combat  service,  and  combat  service  support  reported 
similar  levels  of  concern  and  difficulty  across  the  different  types  of  threats,  with  the  greatest 
concern  being  for  relevant  operational  threats  such  as  Improvised  Explosive  Devices  (IED)  and 
suspicious  persons  and  vehicles.  During  interviews,  Soldiers  discussed  how  they  gather 
information,  spot  cues  and  trends,  scan  environments,  observe  changes,  aggregate  information, 
and  fonn  mental  pictures  of  situations.  Their  descriptions  of  the  types  of  threats  and  cues  present 
in  threat  situations  feed  into  research  stimuli  development  and  informed  the  exercises  Soldiers 
would  complete  in  the  experiments  that  followed.  In  addition,  these  data  contributed  to  the 
development  and  refinement  of  the  threat  detection  loop  that  represents  a  threat  detection  mental 
model  and  illustrates  the  threat  detection  process. 

Phase  III  was  a  preliminary  assessment  of  the  refined  research  materials.  These  materials 
included  exercises  focused  on  dynamic  threat  monitoring/search,  threat  prioritization,  and  causal 
reasoning.  Soldiers  completed  the  exercises  and  provided  feedback  about  the  accuracy  and 
relevance  of  the  materials.  They  discussed  the  photos  presented  during  the  exercises  and 
provided  interpretations  of  the  threats  present  in  the  photos.  They  also  reasoned  about  why  the 
threats  were  important,  rated  the  degree  to  which  the  threats  are  likely  to  come  to  fruition,  and 
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described  how  they  would  handle  similar  threats  based  on  their  experiences.  These  findings,  as 
well  as  feedback  from  the  Soldiers  about  the  exercises,  indicated  that  the  materials  developed 
would  provide  a  good  test  of  threat  detection  skill  and  the  photos  and  exercises  were  relevant  to 
current  threat  detection  activities  in  the  OE. 

This  report  presents  the  findings  from  the  final  research  phase  and  consists  of  a 
discussion  of  the  design  and  development  of  the  training  exemplar,  including  a  formative 
evaluation  and  suggestions  for  refinements  and  extensions  of  the  research  and  training.  The 
report  presents  information  as  follows: 

•  introduction:  the  remainder  of  the  introduction  will  discuss  the  threat  detection  loop  and 
research  on  expertise  development, 

•  research:  including  discussion  of  the  experiment  methods,  procedures,  and  results, 

•  training  design  and  development:  the  design  includes  the  process  for  identifying  training 
stimuli  and  content,  and  the  development  includes  construction  of  the  computer-based 
training  exemplar  and  results  of  two  formative  evaluations,  and 

•  designs  for  refinements  and  extensions:  this  section  suggests  the  use  of  eye-tracking 
technologies  and  immersive  environments  to  expand  our  understanding  of  threat 
detection  by  examining  threat  detection  perfonnance  in  more  realistic  settings  such  as, 
during  field-exercises  and  in  Combat  Training  Centers. 

Threat  Detection  Model 

An  analysis  of  the  literature  and  of  findings  from  the  initial  phases  of  this  research 
(Zimmerman  et  ah,  In  Preparation)  led  to  the  refinement  of  a  threat  detection  loop  that  represents 
the  cyclical  process  engaged  in  during  threat  detection.  Findings  suggested  that  the  primary 
components  of  visual  threat  detection  are  (See  Figure  1): 

•  dynamic  threat  monitoring  or  maintaining  a  vigilant  search  of  the  environment  among 
competing  visual  cues, 

•  threat  prioritization  or  identifying  threat-relevant  versus  threat-irrelevant  cues,  and 

•  causal  reasoning  or  determining  why  certain  cues  are  present  or,  for  instance,  what  the 
enemy  may  interpret  as  a  good  concealment  location. 
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Figure  1.  Threat  Detection  Loop  (inner  rectangles)  and  measurable  behaviors  of  expert  threat 
detection  skill  (outer  rectangles). 


Figure  1  depicts  steps  in  the  threat  detection  loop  (inner  rectangles)  and  their 
corresponding  behaviors  or  abilities  (outer  rounded  rectangles).  For  instance,  to  effectively 
monitor  and  search  activities  (loop)  threat  detectors  need  to  encode  relevant  details  (a).  Based  on 
an  understanding  of  visual  features  related  to  threats,  experienced  threat  detectors  should  be  able 
to  classify  threats  accurately  and  quickly  (b).  The  loop  indicates  that  experienced  threat 
detectors  are  likely  to  see  patterns  during  visual  search  that  match  to  their  prior  experiences,  and 
thus  demonstrate  better  ability  to  locate  and  classify  high  priority  threats  (i).  Their  prior 
experience  should  influence  their  decisions  as  they  determine  an  appropriate  action  (d)  and 
enhance  their  ability  to  detect  changes  and  anomalies  in  their  environments  (g).  Based  on 
experience,  these  threat  detectors  should  also  demonstrate  ability  to  construct  causal  stories 
about  situations  (h)  and  predict  how  the  threats  might  materialize.  This  ability  to  incorporate 
information  and  construct  stories  should  allow  them  to  identify  and  classify  threats  in  a  selective 
manner  (c),  which  would  influence  their  decision  threshold  for  taking  action.  Threat  detector 
ability  to  weight  the  strength  of  the  evidence  regarding  presence  and  severity  of  potential  threats 
(f)  also  influences  this  threshold.  Because  threat  detectors  can  incorporate  their  experiences  as 
they  process  relevant  infonnation  to  identify  threats,  this  model  predicts  they  will  demonstrate 
ability  to  identify  appropriate  courses  of  action  (e). 


The  loop  consists  of  several  processes  that  Soldiers  develop  as  they  acquire  threat 
detection  knowledge  and  experience.  Because  the  research  conducted  in  Phases  I  through  III 
focused  on  exercises  such  as  threat  prioritization  and  causal  reasoning,  the  data  revealed 
informational  cues  and  experiences  that  are  most  important  for  gaining  knowledge  and 
developing  the  ability  necessary  for  visual  threat  detection.  To  identify  the  processes  that  fonn 
the  threat  detection  loop,  we  relied  on  previous  research  regarding  decision-making  models, 
vigilance  and  detection  loops,  and  models  of  expert  decision-making  (Boyd,  1987;  Klein,  1998; 
Klein,  Calderwood  &  Clinton-Cirocco,  1986;  Mueller,  2009;  Mueller  &  Weidemann,  2008). 
The  model  allows  us  to  make  predictions  about  Soldiers’  performance  as  they  gain  competency 
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and  develop  toward  expert  stages.  According  to  the  model,  proficient  Soldiers  should  be  better 
able  to  monitor  and  search  their  environment  in  order  to  detect  changes  or  anomalies. 

One  goal  of  this  research  was  to  identify  differences  in  experienced  and  novice  threat 
detection  processes,  thus  the  model  provided  a  guideline  for  identifying  metrics  of  expertise. 
Each  step  in  the  loop  is  associated  with  measureable  behaviors  that  indicate  the  influence  of 
experience  on  threat  detection.  To  determine  how  behaviors  contribute  to  threat  detection,  it  is 
possible  to  measure  these  behaviors  in  exercises  or  laboratory  settings.  As  experience  increases, 
Soldiers  may  improve  their  ability  to  perfonn  many  of  the  component  skills  required  to  detect 
threats. 


This  research  examined  several  of  the  behaviors  identified  in  this  model  to  detect  the 
presence  of  these  skills  in  more  experienced  Soldiers  and  to  create  training  to  improve  threat 
detection  skills  in  less  experienced  Soldiers.  Given  this  model,  we  expected  to  find  a  difference 
in  the  skills  of  more  experienced  Soldiers  compared  to  the  skills  of  less  experienced  Soldiers. 

For  example,  experienced  Soldiers  should  more  readily  identify  threat  versus  non-threat  features 
compared  to  less  experienced  Soldiers  who  may  not  have  the  experience  and  subsequent  mental 
models  to  understand  which  features  indicate  a  relevant  threat.  It  is  also  likely  that  experienced 
Soldiers  would  make  comments  about  these  features  that  differ  from  comments  made  by  non- 
deployed  Soldiers.  Soldiers  with  little  experience  may  not  be  proficient  at  identifying  cues  or 
features  in  the  environment  that  are  relevant  to  detecting  threats.  They  may  also  have  difficulty 
distinguishing  between  cues  that  are  relevant  and  those  that  are  irrelevant  or  typically  do  not 
indicate  a  threat.  Because  they  lack  the  relevant  experience  base,  they  may  not  be  able  to  explain 
why  certain  cues  are  threats  in  specific  contexts  or  hypothesize  about  what  the  enemy  might  do 
in  similar  circumstances.  Experienced  Soldiers  should  be  able  to  reason  through  why  features 
indicate  a  threat  and  describe  hypothetical  actions  and  outcomes. 

Expertise  Development 

The  observations  of  police  officers  in  Phase  I  indicated  that  the  less  experienced  officer 
tended  to  respond  readily  to  many  potential  threats  without  discriminating  between  cues  based 
on  threat  severity  or  likelihood.  Meanwhile,  more  experienced  officers  seemed  to  evaluate 
possible  threats  with  little  deliberation  and  judge  threat  severity  by  weighing  the  probability  of 
effective  outcomes.  Research  shows  that,  compared  to  their  novice  counterparts,  domain  experts 
are  better  able  to  perceive  and  process  infonnation,  recognize  pertinent  cues,  and  match  those 
present  cues  to  previous  experiences  in  a  manner  that  facilitates  successful  action  (Ericsson  & 
Charness,  1994;  Goodrich,  Sterling,  &  Boer,  2000).  While  the  initial  research  phases  indicated 
that  experience  and  knowledge  influence  and  facilitate  the  threat  detection  process,  that  research 
did  not  compare  the  cognitive  skills  of  Soldiers  with  varying  degrees  of  threat  detection 
experience.  In  order  to  create  effective  training  that  leveraged  Soldiers’  experiences,  it  was 
important  to  determine  if  differences  in  cognitive  and  perceptual  strategies  and  skills  exist. 

A  training  program  that  improves  the  cognitive  skills  used  to  detect  threats  should 
incorporate  exercises  that  promote  novice  advancement  toward  expert  perfonnance  levels. 
Dreyfus  and  Dreyfus  (1986)  proposed  a  five-stage  model  of  skill  acquisition  to  describe  how 
decision-making  changes  as  experience  increases.  At  the  novice  stage  of  this  model,  decision 
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makers  recognize  objective  facts  about  situations  and  apply  “context-free”  rules  to  the  situation. 
As  decision  makers  progress  through  the  advanced  beginner  and  competent  stages,  they  begin  to 
recognize  meaningful  elements  of  specific  situations  and  begin  to  incorporate  those  elements  into 
decisions.  As  expertise  develops  through  the  proficient  stage,  decision  makers  begin  to  perceive 
situations  as  a  whole,  rules  become  less  important,  and  decision  makers  become  more  flexible 
and  react  faster  to  incoming  information.  In  the  final  or  expert  stage,  decision  makers  intuitively 
recognize  situations  and  match  them  to  previous  experiences.  They  use  mental  simulation  to 
predict  events  and  outcomes  of  actions  and  they  deal  with  uncertainty  by  story-building  and 
actively  seeking  infonnation  (Dreyfus  &  Dreyfus,  1986;  Ross,  Phillips,  Klein,  &  Cohn,  2005). 
This  model  provides  a  template  for  how  threat  detection  proficiency  may  change  as  Soldiers 
acquire  experience.  The  behaviors  predicted  by  the  threat  detection  loop  (presented  in  Figure  1) 
are  characteristic  of  decision  makers  in  the  proficient  and  expert  stages  of  the  Dreyfus  and 
Dreyfus  model. 

Previous  research  examining  assessments  made  by  novice  and  experienced  police  officers 
of  a  traffic  stop  video  provided  insight  into  the  differences  in  perceptions  and  interpretations  of  a 
threatening  event  (Zimmerman,  2008).  In  that  research,  police  officers  viewed  a  video  taken 
from  a  patrol  car  dashboard  camera  in  three  segments.  At  the  end  of  each  segment,  they 
provided  their  assessment  of  the  situation  including  cues  that  indicated  a  threat,  interpretations  of 
the  events,  and  courses  of  actions  they  would  take.  Novice  and  experienced  police  officers 
tended  to  focus  on  the  same  types  of  cues,  but  differed  in  how  they  elaborated  on  their 
observations.  Consistent  with  prior  research  (Klein,  Phillips,  Battaglia,  Wiggins,  &  Ross,  2002; 
Phillips,  Klein,  &  Sieck,  2004),  experienced  police  officers  provided  descriptions  of  the  event 
that  were  more  elaborate,  provided  and  interpreted  more  cues,  and  made  more  interpretations  and 
predictions  about  what  would  occur.  This  finding  is  similar  to  previous  research  showing  that 
less  and  more  experienced  decision  makers  use  similar  sensemaking  strategies,  but  experts  show 
a  deeper  understanding  of  the  situation  and  provide  richer  explanations  (Klein  et  ah,  2002). 
Novice  officers  tended  to  focus  on  procedural  issues,  such  as  officer  safety,  and  they  rarely  tried 
to  ascribe  meaning  to  threatening  behaviors  or  interpret  the  scene  in  multiple  ways.  Experienced 
officers  tended  to  consider  the  context  surrounding  the  situation  and  make  interpretations  based 
on  their  prior  knowledge  of  officer,  driver,  and  criminal  behavior.  They  weighed  the  relevance 
and  importance  of  cues  within  the  context  of  the  current  situation  whereas  novice  police  officers 
identified  threat  cues  and  based  their  probable  actions  on  the  cues  without  necessarily 
considering  the  context. 

In  Zimmerman,  et  ah,  (In  Preparation),  Soldiers  with  combat  experience  in  the  OE 
emphasized  the  importance  of  experience  when  searching  environments  to  detect  threats.  They 
reported  threat  search  processes  that  focused  on  procedural  tasks,  such  as  using  intelligence  and 
typical  infonnation  gathering  techniques,  but,  unlike  the  novice  police  officers,  they  did  not  tend 
to  focus  on  Soldier  safety.  Instead,  they  provided  examples  from  their  experiences  of  techniques 
they  use  to  search  for  threat  cues,  such  as  noticing  atypical  threat  indicators  and  leveraging  the 
opinions  of  other  Soldiers.  Soldiers  were  able  to  verbalize  the  learning  process  that  occuned 
during  their  deployment,  particularly  how  they  learned  to  sort  and  filter  infonnation  and 
distinguish  between  threats  and  non- threats.  Their  comments  also  indicated  that  they  could 
process  threat  cues,  predict  the  consequences  of  those  cues,  and  take  preventative  actions.  They 
also  recognized  that  some  threat  cues  were  too  prevalent  or  random  to  spend  much  energy 
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detecting,  such  as  trash  along  the  side  of  the  road.  Soldiers’  comments  indicated  that  they 
consider  their  surroundings  when  interpreting  threat  cues,  for  instance,  by  noticing  trends  and 
changes  and  reassessing  situations  given  the  new  context.  The  Soldiers  in  Phase  III  viewed 
photos  and  not  only  provided  comments  about  the  threat  cues  present  but  also  were  able  to 
present  detailed  hypothetical  scenarios  based  on  the  scenes  in  the  photos  and  discuss  plausible 
actions  they  would  take.  These  comments  are  characteristic  of  experience-based  processing  and 
interpretations  of  situations. 


Experiment 


Overview 

Based  on  the  expert  threat  detection  model  (Figure  1)  and  the  results  from  the  previous 
experiments,  subsets  of  the  stimuli  were  piloted  during  the  Phase  III  research  (Zimmerman,  et 
ah,  In  Preparation).  These  stimuli  provided  an  opportunity  to  identify  useful  targets  for  use  in 
the  training  exemplar  development.  The  current  research  added  a  change  detection  exercise  to 
investigate  differences  in  the  ability  to  detect  changes  based  on  experience  and  to  understand  the 
extent  that  Soldiers  use  change  detection  to  identify  threats.  During  the  initial  phases  of  this 
research,  Soldiers  indicated  that  detecting  changes  is  an  important  component  of  threat  detection, 
as  represented  in  the  threat  detection  loop. 

Soldiers  at  Fort  Hood,  TX,  engaged  in  three  computer  exercises:  a  prioritized  threat 
search,  dynamic  threat  detection,  and  change  detection.  Upon  completion  of  those  exercises,  the 
Soldiers  viewed  photos  during  interviews  and  identified  threat  cues,  discussed  these  cues  and 
areas  of  concern,  and  provided  their  interpretation  of  the  scenes. 

Method 

Participants. 

Forty-seven  U.S.  Anny  Soldiers  participated  in  computer  exercises  and  interviews. 
Twenty  Soldiers  were  NCOs  (Sergeant;  SGT  to  Master  Sergeant;  MSG),  one  was  an  Officer 
(First  Lieutenant;  1LT),  and  26  were  enlisted  (Private;  Private  to  Corporal;  CPL/  Specialist; 
SPC).  All  Soldiers  were  male.  Table  1  lists  the  average  age,  mean  time  in  service,  and  the  mean 
number  of  times  deployed  for  each  group. 

Table  1 

Junior  enlisted  and  NCO  mean  age  (range),  time  in  service  (range),  and  number  of 
deployments 


Rank 

Mean  Age 

Mean  Time  in  Service 

%  Deployed 

Junior  Enlisted 

23  (18-37) 

27  mo.  (4-72) 

54% 

NCO/Officer 

29  (23-44) 

103  mo.  (31-192) 

95% 
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Soldiers  reported  zero  to  four  deployments.  Thirty-four  Soldiers  reported  deploying  at 
least  once.  Of  those  who  deployed,  83%  were  combat  arms.  Table  2  lists  the  number  of  Soldiers 
in  each  rank  by  the  number  of  times  they  have  deployed. 

Table  2 

Soldier  rank  and  number  of  deployments 


Rank 

Total 

0 

Number 

1 

of  Deplo 
2 

yments 

3 

4 

Private 

2 

1 

1 

PV2 

3 

3 

PFC 

11 

8 

3 

CPL/SPC 

10 

7 

3 

SGT 

1 

1 

SSG 

11 

1 

1 

6 

3 

SFC 

7 

3 

3 

1 

MSG/1  SG 

1 

1 

1LT 

1 

1 

Total 

47 

13 

14 

12 

7 

1 

The  most  common  military  occupational  specialty  (MOS)  reported  by  Soldiers  was 
Infantryman  (1  IB),  with  15  deployed  and  2  non-deployed  Soldiers  reporting  this  MOS  (Table  3). 
Only  one  Soldier  reported  a  current  MOS  different  from  the  MOS  held  while  deployed.  This 
Soldier  reported  that  his  current  MOS  was  Armor  Officer  (19A),  however,  his  MOS  while 
deployed  was  a  Motor  Transport  Operator  (88M). 

Table  3 

Soldier  MOS  and  deployment  status 


Deployed 

MOS 

Yes 

No 

1  IB  Infantryman 

15 

2 

12B  Combat  Engineer 

5 

2 

13F  Field  Artillery  Forward  Observer 

3 

1 9A  Armor  Officer 

1 

19D  Armor  Scout 

2 

1 

19KM-1  Crewman 

4 

2 

25N  Nodal  Network  Operator  (Signal) 

25Q  Multichannel  Transmission  Systems  Operator- 

1 

T. 

Maintainer 

74D  Chemical,  Biological,  Radiological,  and  Nuclear 
Warfare  Specialist 

1 

91AM-1  Tank  Mechanic 

1 

91M  M-2/3  Bradley  Mechanic 

1 

1 

92F  Petroleum  Supply  Specialist 

2 

Total 

34 

13 
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Deployed  Soldiers  rated  how  often  they  went  ‘outside  the  wire,’  or  how  often  they 
traveled  off  their  Forward  Operating  Base  (FOB)  or  similar  location,  on  a  scale  from  1  (rarely  or 
never)  to  4  (almost  every  day).  Twenty-seven  Soldiers  reported  going  outside  the  FOB  almost 
every  day  (M=  3.65,  SD  =  0.77). 

Materials. 

Computer-controlled  Questionnaire.  Materials  similar  to  those  used  in  the  previous 
experiments  were  used  for  the  current  research  (see  Zimmerman  et  al.,  Appendix  A).  These 
included  a  computer-controlled  demographic  and  experience  questionnaire  that  contained  a 
subset  of  the  questions  asked  in  the  previous  research  and  additional  questions  relevant  to  the 
current  research  (Appendix  A).  Special-purpose  software  using  the  Psychology  Experiment 
Building  Language  (PEBL)  0.09  computer  experimentation  system  controlled  the  computer 
exercises  (Mueller,  2009).  The  participant  section  (above)  presents  the  relevant  data  from  this 
questionnaire. 

Computer-controlled  Imagery  Analysis.  The  three  computer-controlled  threat  detection 
exercises  presented  photos  selected  from  a  set  of  48  images  approved  for  public  release.  These 
images  were  retrieved  from  three  main  sources,  www.Defensehnagery.mil, www.Flickr.com, 
and  www.defense.gov.  Findings  from  the  previous  experiments  guided  the  photo  selection 
process  for  the  final  set  of  40  images.  These  images  were  divided  into  four  groups  of  10  and 
each  group  of  10  images  was  assigned  to  one  of  four  conditions:  time-unlimited  prioritized 
threat  search,  time-limited  prioritized  threat  search,  single-task  threat  detection,  and  dual-task 
threat  detection.  Four  counterbalanced  conditions  were  devised  using  a  Latin  square  design  and 
participants  were  assigned  to  each  counterbalanced  condition  based  on  the  order  in  which  they 
participated  in  the  data  collection. 

Twelve  additional  images  were  selected  as  stimuli  for  the  change  detection  exercise. 
Image-editing  software  enabled  the  creation  of  two  versions  of  each  image  by  seamlessly  adding 
or  removing  specific  features  from  the  images.  Changes  included  both  threat-relevant  features 
(brickwork,  trash,  people,  vehicles,  etc.),  and  threat-irrelevant  features  (the  length  of  poles, 
ornamental  architecture  detail,  etc.).  Soldiers  viewed  one  version  of  each  of  the  change- 
detection  images  during  the  initial  prioritized  threat  search  exercise  and  tried  to  detect  changes 
on  the  second  version  of  each  image  at  the  end  of  the  sessions. 

Interview  Protocol.  The  interviews  focused  on  gathering  infonnation  about  the  causal 
reasoning  used  to  make  threat  detection  choices  during  the  computer  exercise.  The  set  of  photos 
used  in  the  computer  exercises  and  previous  research  were  used  to  gather  information  about 
Soldier  thought  processes  and  reasoning  while  detecting  threats. 
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Procedure. 


The  research  consisted  of  two  main  parts:  a  computer-controlled  exercise  and  a  semi- 
structured  interview.  At  the  start  of  each  session,  following  administration  of  privacy  act  and 
informed  consent  protocols,  Soldiers  read  a  description  of  their  exercises  and  were  told  that  they 
were  there  to  provide  information  about  their  threat  detection  experiences  and  to  identify 
common  threat  detection  situations,  indicators,  and  challenges  in  the  OE.  Soldiers  then 
completed  the  computer  exercises  at  their  own  pace  which  typically  took  45  minutes  to 
complete. 

Following  the  demographic  questions,  Soldiers  completed  five  consecutive  computerized 
exercises  developed  to  measure  attention,  search,  reasoning,  and  prioritization  strategies  for 
threat  detection.  The  first  two  exercises  involved  a  resource-limited  threat  search  process 
referred  to  as  the  prioritized  threat  search.  The  second  two  exercises  involved  dynamic  threat 
detection  in  which  Soldiers  engaged  in  a  stimulus  detection  exercise  with  distracters.  Finally,  in 
the  change  detection  exercise  Soldiers  viewed  changed  imagery  which  they  first  saw  during  the 
prioritized  threat  search  exercise  and  they  annotated  any  changes  observed. 

Prioritized  threat  search.  The  first  two  computer-controlled  exercises  following  the 
demographic  questionnaire  required  Soldiers  to  search  an  image  for  potential  threat  targets  using 
a  virtual  “scope”  (Figure  2).  Each  image  contained  approximately  10  possible  target  locations. 
The  annotations  provided  by  Soldiers  in  the  previous  experiments  formed  the  basis  for  these 
target  locations.  The  click  locations  from  previous  experiments  were  not  suitable  for  direct  use 
because  they  generally  contained  too  many  targets,  thus  a  small  number  of  reasonable  and 
distinct  target  locations  was  selected  based  on  visual  examination  of  the  annotated  imagery. 
When  Soldiers  clicked  the  100-pixel  diameter  circular  scope  on  one  of  the  target  locations,  a  red 
dot  (identified  target)  would  display  with  a  65%  probability  rate  of  detection.  This  probability 
allowed  participants  to  conduct  repeated  searches  of  the  image  in  locations  they  suspected  of 
containing  a  target.  For  each  image,  Soldiers  could  make  a  maximum  of  10  search  clicks.  The 
first  search  exercise  had  no  time  limit  and  contained  18  images.  The  second  search  exercise  also 
had  18  images.  Soldiers  had  a  maximum  of  one  second  per  click  to  search  each  photo  (if  they 
did  not  use  the  click  after  one  second  it  would  become  unavailable)  and  after  each  click  the  timer 
decremented  to  the  next  even  remaining  second.  This  scheme  encouraged  Soldiers  to  search 
steadily  throughout  the  lOsecond  time  period  rather  than  click  haphazardly  when  time  was  about 
to  elapse.  Each  condition  in  the  prioritized  threat  search  exercise  contained  12  images  from  the 
main  set  of  counterbalanced  photos  and  an  additional  six  images  used  in  the  first  part  of  the 
change  detection  exercise.  Soldiers  did  not  know  that  the  change-detection  images  would  appear 
again  later. 
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Figure  2.  Screenshot  of  prioritized  threat  search  exercise. 1 


Find  hidden  targets  with  microwave  scope. 


10— 
9  — 
8  — 
7  — 
6  — 


Try  to  find  as  many  hidden 
targets  as  you  can.  The  scope 
is  not  perfect,  so  it  may  miss 
some  targets  and  you  may 
need  to  look  in  the  same 
place  more  than  once.  You 
will  be  given  as  much  time  as 
you  need  per  click,  so  search 
as  efficiently  as  you  can. 


Dynamic  threat  detection.  The  design  of  the  cued  threat  detection  exercise  allowed 
experienced  Soldiers  to  use  their  background  knowledge  of  threats  to  guide  their  visual  attention. 
On  each  trial,  the  screen  displayed  an  image  overlaid  with  75  transparent  grey  dots  (noise) 
distributed  randomly,  but  consistently  over  the  image.  These  dots  vibrated  and  obscured  the 
image.  These  grey  dots  obscured  the  appearance  of  the  target  dot  and  required  Soldiers  to 
maintain  constant  vigilance  in  order  to  detect  the  target  among  the  field  of  dots.  The  target  dot 
was  identical  to  the  noise  except  that  it  had  a  light  orange  rim  around  the  circle.  The  Soldiers’ 
task  was  to  click  the  mouse  button  as  soon  as  a  target  was  detected  which  revealed  a  mouse 
cursor.  Once  the  cursor  appeared,  they  could  indicate  the  location  of  the  target.  Each  image 
presented  seven  targets  in  threat-consistent  locations  and  three  targets  in  threat-inconsistent 
locations.  The  threat-consistent  locations  were  determined  by  examining  responses  from 
previous  research  with  the  same  or  similar  imagery. 


1  The  red  bar  on  the  right  of  image  shows  remaining  search  clicks  available.  Found  targets  are  displayed  as 
transparent  red  dots  and  the  virtual  threat  scope  as  a  transparent  green  overlay. 

2  Each  transparent  dot  had  a  radius  of  12  pixels.  Throughout  the  trial,  the  location  of  these  dots  was  jittered  with  a 
bivariate  uniform  distribution  of  five  pixels.  The  dot  locations  updated  approximately  every  100  milliseconds.  A 
target  dot  appeared  in  the  dot  field  at  random  delays  of  between  5  and  10  seconds. 
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Watch  the  image  for  threats. 

Watch  the  scene  for  a  target  and  the 
lower  right  console  for  at  least  three  red 
lights.  If  you  see  an  orange-outlined 
target  on  the  image,  or  three  or  more 
lights  turn  red  on  the  console,  click  the 
mouse  button  to  display  the  cursor. 

Then,  click  on  the  location  of  the  change. 
Click  the  'play'  button  to  begin. 

Watch  for  target  on  light  panel. 


T, 

Figure  3.  Screenshot  of  dynamic  threat  detection  exercise. 

A  second  round  of  the  dynamic  threat  detection  exercise  required  simultaneous 
monitoring  of  changes  occurring  on  a  secondary  light  monitor.  The  light  monitor  was  located  in 
the  lower  right  section  of  the  image  and  had  four  virtual  LED  lights  that  flashed  either  blue  or 
red  (Figure  3).  On  each  cycle,  three  lights  would  illuminate,  typically,  one  would  be  red  and  two 
would  be  blue.  When  the  target  appeared,  three  of  the  four  lights  would  flash  red,  but  would  still 
update  on  the  same  cycle  so  the  three  red  lights  would  appear  for  the  same  length  of  time  as  the 
other  combinations  of  blue  and  red  lights.  Soldiers  needed  to  monitor  both  the  photo  for  the 
target  (grey  dot  with  orange  outline)  and  the  light  display  for  three  red  lights.  When  they 
detected  the  three  red  lights,  they  would  click  on  the  screen  outside  the  photo  rather  than  on  the 
location  of  the  target  location  in  the  photo.  The  purpose  of  this  manipulation  was  to  assess  the 
degree  that  threat  detection  experience  affected  performance  when  simultaneously  performing 
multiple  tasks.  Research  on  novice  and  expert  performance  shows  that  novices  typically  must 
consciously  attend  to  the  task  they  are  performing  while  experts  do  not  need  to  apply  as  much 
conscious  thought  and  attention  (Beilock  &  Carr,  2004;  Beilock,  Wierenga,  &  Carr,  2002; 
Shiffrin  &  Schneider,  1977).  As  a  result,  experts  should  be  able  to  devote  more  attention  to  the 
dual  task  without  a  performance  decrement  while  novices  should  perfonn  poorly  under  dual  task 
conditions. 

Change  detection.  The  final  computer  exercise  involved  “change  detection,”  in  which 
Soldiers  viewed  altered  versions  of  12  images  they  had  seen  during  the  prioritized  threat  search 
exercise.  They  viewed  half  of  the  images  in  the  untimed  version  of  the  exercise  and  the  other 
half  in  the  timed  version  with  the  photos  counterbalanced  across  Soldiers.  In  addition,  the  first 


3  At  this  point,  three  of  the  secondary  monitor  lights  were  illuminated. 
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version  of  the  change  photos  shown  appeared  at  random  so  that  half  the  Soldiers  saw  the  original 
image  during  the  exercise,  whereas  the  other  half  saw  the  altered  version.  The  changed  images 
were  intermixed  randomly  with  the  normal  threat  search  imagery,  thus  Soldiers  had  no  indication 
that  these  images  would  later  appear  as  a  test  of  their  change  detection  skill.  At  test,  Soldiers 
were  told  that  several  things  were  different  about  the  image  compared  to  the  first  time  they  had 
seen  it  (Figure  4).  They  used  the  mouse  to  indicate  a  location  and  gave  a  brief  annotation 
describing  what  they  believed  had  changed. 


Type  note.  Hit  enter  to  finish  note 


You  have  seen  the  following 
image  before.  However, 
several  things  about  the 
picture  are  different.  Click  on 
locations  you  suspect  to  be 
different,  and  identify  what 
differs. 


NEXT 

1  pile  of  bricks  got  larger 


is  filled  inj 


Figure  4.  Screenshot  of  change  detection  exercise. 


Interviews.  After  the  computer  exercise,  the  research  staff  interviewed  Soldiers  about 
their  threat  detection  experiences.  The  interviews  lasted  approximately  30  minutes.  Soldiers 
were  instructed  to  look  through  a  set  of  images  (that  had  been  used  in  the  computer  tests)  and 
select  one  or  more  that  reminded  them  of  a  situation  they  had  experienced  (in  training  and/or 
operational  environments).  They  identified  and  described  the  threats  in  each  photo  and  provided 
explanations  about  the  threats  they  deemed  important.  They  discussed  the  threat-related  events 
that  might  occur  in  each  photo  and  discussed  possible  threat  scenarios.  Finally,  Soldiers 
provided  feedback  about  the  computerized  exercises  and  made  suggestions  for  computer-based 
training  exemplar  development.  During  the  debriefing,  Soldiers  could  ask  questions  and  provide 
feedback  about  the  experiment  and  the  project.  They  received  contact  information  for  mental 
health  assistance  if  discussion  of  their  experiences  brought  up  any  negative  feelings  or  memories. 
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Participants  indicated  locations  of  changes  and  annotated  what  they  believed  the  change  was. 
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Analysis. 


Threat  Detection  Exercises.  The  R  statistical  computing  platform  Version  2.10.1  (R 
Core  Development  Team,  2009)5  was  used  to  analyze  the  data  from  the  computerized  exercises. 
R  provided  a  number  of  useful  add-on  modules  that  enabled  image  analysis  and  display.  In 
addition,  for  the  prioritized  threat  search  exercise,  R  created  images  that  overlaid  transparent  red 
indicators  on  top  of  the  original  stimuli  to  indicate  the  clicked  locations  of  all  participants 
(Appendix  B). 

Interviews.  To  conduct  the  interview  analysis,  interview  recordings  were  transcribed. 

The  Soldiers’  comments  were  sorted  into  pre-detennined  categories,  which  had  been  developed 
during  the  previous  phases  of  research  (Zimmerman  et  ah,  In  Preparation).  These  categories 
include  types  of  threats,  threat  cues,  strategies  for  threat  detection,  threat  detection  tasks,  skills, 
challenges,  and  training  (Table  4).  Because  Soldiers  with  different  amounts  of  experience 
commented  on  the  same  photos,  we  compared  comments  made  by  less  and  more  experienced 
Soldiers  in  each  of  these  categories  and  compared  comments  made  by  multiple  Soldiers  to  the 
same  photos.  Interviews  were  coded  line  by  line  to  identify  potentially  important  pieces  of  data. 
We  compiled  these  specific  comments  according  to  these  categories  and  kept  the  coder  blind  to 
each  Soldier’s  characteristics.  After  comments  were  sorted  into  each  broad  category,  specific 
comments  were  separated  out  according  to  finer  distinctions.  For  example,  within  the  category 
threat  cues,  comments  could  be  those  related  to  behavioral,  person,  or  environmental  cues. 

After  all  comments  were  coded  according  to  these  categories,  we  began  to  identify 
elements  in  the  5-Stage  Skill  Acquisition  Model  (Dreyfus  &  Dreyfus,  1986).  Each  comment  was 
represented  at  a  stage  in  the  model  so  we  could  ascertain  their  level  of  skill  development.  At  that 
point,  we  identified  Soldiers’  experience  levels  and  identified  comments  as  belonging  to 
inexperienced  Soldiers  (no  deployments)  or  experienced  Soldiers  (Soldiers  with  one  deployment, 
and  Soldiers  who  had  between  two  and  four  deployments).  The  number  of  Soldiers  who  made 
comments  in  each  of  the  high-level  categories  were  counted  and  separated  according  to 
experience  level.  After  that,  comments  were  separated  within  each  category  according  to 
experience  level  so  that  similarities  and  differences  could  be  examined  within  the  level  of 
experience. 

In  order  to  identify  comments  as  they  related  to  the  threat  detection  model,  we  relied  on 
the  same  initial  coding  scheme,  although  this  time  solely  focusing  on  the  experienced  Soldiers. 
We  then  identified  each  comment  according  to  the  threat  detection  model  and  sorted  the 
comments  according  to  each  part  of  the  model. 


5  R  is  a  commonly  used  statistical  computing  tool  available  for  use  on  all  common  computing  platforms, 
http://www.r-project.org/ 
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Table  4 


Interview  data  categories 


Interview  Data  Classification 

High-level  Categories 

Types  of  Threats 

Enemy  activity,  enemy  tactics,  activity  in  the 
environment,  troop  activity,  crowd  behavior 

Threat  Cues 

Behavioral  cues,  environmental  cues, 
physical  cues  (such  as  wires,  garbage), 
patterns  of  cues 

Strategies  for  Threat  Detection 

Tactics,  information  gathering,  spot  cues,  spot 
trends,  ask  what-if  questions  while  assessing 
situations 

Threat  Detection  Tasks 

Scan  environment,  question  subjects,  active 
evidence  search,  observe  situations  and 
environments 

Threat  Detection  Skills 

Think  like  the  enemy,  prioritize  infonnation, 
aggregate  infonnation,  form  mental  picture  of 
situation,  regularly  ask  what-if  questions 

Challenges  in  Detecting  Threats 

Noticing  cues,  noticing  patterns,  enemy 

Tactics,  Techniques,  and  Procedures  (TTP), 
information  reliability,  night  operations 

Solutions  for  Threat  Detection 

T  raining/Preparation 

Scenario-based  training,  train  consequences, 
use  Iraqi/Afghan  role-players/environments, 
think  like  the  enemy,  real-world  experience, 
train  with  what-if  questions 

Results  and  Discussion 

Threat  Detection  Exercises. 

Prioritized  Threat  Search.  The  prioritized  threat  search  exercise  measured  performance 
with  and  without  time  pressure.  The  number  of  threats  discovered  by  Soldiers  provided  a  metric 
of  performance  in  both  conditions.  Soldiers  found  targets  in  an  average  of  2. 14  clicks  while 
under  time  pressure  and  2.32  clicks  when  not  under  time  pressure.  A  Welch  2-sample  t-test 
determined  these  to  be  significantly  different  [7(1374)  =  2.05, p  <  .05,  Cohen ’s  d=  0.1 1], 
showing  that  time  pressure  reliably  reduced  the  number  of  targets  found.  Another  way  to  judge 
search  accuracy  was  to  compare  each  Soldier’s  click  to  the  other  Soldiers’  click  locations.  For 
each  click,  the  number  of  clicks  made  by  other  Soldiers  that  were  within  the  scope  radius  of  the 
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Mean  number  ol clicks 


specified  location  was  counted.  This  value  ranged  from  0  to  77  with  a  mean  of  18.4.  The 
number  of  clicks  within  the  scope  radius  indicated  whether  the  location  was  in  a  sparse  or  dense 
region.  Sparse  regions  contained  fewer  than  10  close  clicks  indicating  few  Soldiers  clicked  in 
those  regions  while  dense  regions  contained  10  or  more  close  clicks  indicating  many  Soldiers 
clicked  in  those  regions.  Approximately  65%  of  the  click  locations  were  in  dense  regions.  For 
each  Soldier,  the  number  of  clicks  made  in  dense  regions  with  and  without  time  pressures  were 


Figure  5.  Two  measures  of  search  accuracy,  plotted  by  time  pressure  and  experience. 

calculated.  Under  time  pressure,  Soldiers  made  5.96  dense  clicks  on  average.  Under  no  time 
pressure,  they  made  an  average  of  7. 1  dense  clicks.  This  difference  was  statistically  significant 
and  indicated  a  medium-sized  effect,  t(1351)  =  8.90 ,p  <  .01,  Cohen ’s  d=  .48. 6  In  contrast,  there 
was  no  reliable  difference  between  the  mean  number  of  sparse  clicks  in  the  two  conditions 
without  time  pressure  (M  =  2.90  versus  with  time  pressure  (M=  3.07),  t  (1363)  =  -1.4,/?  =  .17, 
Cohen ’s  d=  .07). 

Analysis  of  Soldier  experience  focused  on  the  number  of  deployments.  Figure  5  depicts 
the  relationship  between  experience  and  time  pressure  for  clicks  in  the  predetermined  threat 
regions  and  clicks  in  the  dense  regions.  The  results  show  very  little  difference  across  experience 
levels,  but  highlight  the  impact  that  time  pressure  has  on  performance.  One  explanation  for  the 
relatively  larger  impact  of  time  pressure  on  clicks  in  dense  regions  is  that  when  under  time 
pressure  Soldiers  missed  opportunities  to  search  the  photo  as  time  elapsed.  Similar  results  were 
found  when  using  other  measures  of  performance  such  as  the  number  of  threats  identified. 

Click  density  measures  were  also  used;  those  measures  incorporated  experience  or  how  well  an 
individual  matched  the  group.  Click  density  is  the  most  appropriate  measure  because  there  were 


6  A  Cohen ’s  d  of  .48  indicates  a  medium  effect  size.  Effect  sizes  indicate  the  strength  of  the  relationship  between 
two  means.  When  calculating  Cohen ’s  d,  a  small  effect  is  around  .2,  medium  effect  around  .5,  and  large  effect  .8. 
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no  objectively  correct  threats  in  the  image  and  thus  examining  a  sparse  location  could  be 
considered  incorrect.  Random  clicking  on  an  image  suggests  an  inability  to  recognize  and 
indicate  relevant  threats  whereas  condensed  clicking  suggests  a  focused  search  of  an  image  for 
certain  cues. 

To  detennine  if  the  effects  of  experience  on  search  accuracy  were  reliable,  a  linear 
regression  was  computed  with  linear  predictors  for  number  of  deployments,  time  pressure,  and  a 
deployment  x  time-pressure  interaction.  When  analyzing  the  number  of  clicks  in  which  Soldiers 
found  a  target,  the  number  of  deployments  was  unrelated  to  performance,  but  time  pressure  and 
the  time  pressure  x  deployments  interaction  were  both  statistically  significant  (Table  5).  This 
interaction  shows  that  the  deployment  slope  is  slightly  positive  (.048  clicks/deployment)  when 
under  time  pressure,  but  slightly  negative  (-.118  clicks/deployment)  when  not  under  time 
pressure.  This  result  indicates  that  Soldiers  with  more  deployments  performed  slightly  better 
when  under  time  pressure  versus  when  not  under  time  pressure.  However,  the  results  are  quite 
small  overall.  Most  Soldiers  had  fewer  than  two  deployments  so  the  effect  should  be  interpreted 
with  caution. 

Table  5 

Linear  regression  of  clicks  on  predetermined  targets  by  deploymen  ts  and  time  pressure 


Predictor 

Estimate 

S.E. 

t 

Pr(>|t|) 

Intercept 

2.362 

0.093 

25.278 

<.001 

Deployments 

-0.035 

0.053 

-0.655 

0.512 

Time  pressure 

-0.397 

0.132 

-3.002 

0.003 

Deployments  x  pressure 

0.166 

0.075 

2.194 

0.028 

Residual  standard  error:  1.574  on  1372  degrees  of  freedom 
Multiple  R-squared:  0.007698,  Adjusted  R-squared:  0.005529 
F-statistic:  3.548  on  3  and  1372  df,  p-value:  0.01405 

The  analysis  of  clicks  in  dense  regions  showed  a  reliable  positive  relationship  between 
the  number  of  deployments  and  time  pressure,  but  the  impact  of  time  pressure  was  quite  large 
(by  more  than  one  click).  Table  6  shows  the  linear  regression  results  of  the  model  which  indicate 
only  a  statistically  significant  effect  of  time  pressure.  Together,  these  results  indicate  that  time 
pressure  had  an  impact  on  performance.  The  data  also  suggest  this  impact  might  be  larger  for 
novices  and  experience  reduced  or  even  reversed  this  effect.  Future  research  could  investigate 
this  effect  to  understand  whether  and  why  time  pressure  may  have  potentially  less  influence  on 
experienced  Soldiers.  For  instance,  more  experienced  Soldiers  are  more  likely  able  to  sift 
through  relevant  and  irrelevant  information  even  in  time-pressured  situations. 
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Table  6 


Linear  regression  of  density  by  deployments  and  time  pressure 


Predictor 

Estimate 

S.E. 

t 

Pr(>|t|) 

Intercept 

6.999 

0.141 

49.64 

<.001 

Deployments 

0.078 

0.080 

0.97 

0.335 

Time  pressure 

-1.117 

0.199 

-5.60 

<.001 

Deployments  x  pressure 

-0.018 

0.114 

-0.16 

0.874 

Residual  standard  error:  2.375  on  1372  degrees  of  freedom 
Multiple  R-squared:  0.05566,  Adjusted  R-squared:  0.05359 
F-statistic:  26.95  on  3  and  1372  df,  p-value:  <  2.2e-16 


Dynamic  threat  detection.  The  threat  search  exercise  involved  searching  for  targets  that 
appeared  in  a  dynamic  field  of  transparent  dots  that  jittered  throughout  the  search  (Figure  3). 
Soldiers  responded  correctly  when  they  clicked  the  mouse  while  the  target  appeared  on  the 
screen  and  then  indicated  the  location  of  the  target  within  100  pixels  of  its  actual  location.  While 
100-pixel  criterion  is  somewhat  arbitrary,  Soldiers  were  able  to  meet  this  criterion  the  majority 
of  the  time.  Out  of  5,734  total  trials  in  which  a  target  appeared  on  the  image,  Soldiers  clicked 
the  mouse  to  indicate  the  target  was  present  3,107  times  and  2,598  of  these  times  Soldiers 
indicated  the  target  location  within  100  pixels.  The  number  of  correct  and  precise  trials  only 
varied  from  2,584  to  2,601  when  the  criterion  varied  from  50  to  150  pixels,  indicating  the  100- 
pixel  criterion  was  robust. 

When  analyzing  the  proportion  of  correct  (clicked  mouse  when  saw  target)  and  precise 
(accurately  indicated  location  of  target)  trials  in  the  single  and  dual-task  search  exercises,  the 
results  showed  an  overall  accuracy  rate  of  about  50%.  Fewer  than  half  the  threat-irrelevant 
targets  detected  and  greater  than  half  the  threat-relevant  targets  detected  (Table  7).  Dual  task 
trials  had  higher  mean  accuracy  than  single-task  trials;  however,  this  finding  may  stem  from 
practice  effects  because  the  single  task  was  always  completed  first  and  the  dual-task  second.  We 
intentionally  did  not  counterbalance  the  order  of  the  single  and  dual  task  conditions  because  the 
dual  task  condition  would  have  been  too  difficult  and  the  secondary  task  did  not  require  Soldiers 
to  have  familiarity  with  the  primary  task.  The  photos  were  different  in  the  single  and  dual  tasks. 
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Table  7 


Mean  accuracy  and  time  for  the  threat  search  exercise 


Condition 

Irrelevant  Relevant 

Monitor 

Accuracy 

Single 

0.446  (.026)*  0.585  (.028) 

NA 

Dual 

0.513  (.037)  0.629  (.042) 

0.373  (.057) 

Time  to  respond  (ms) 

Single 

4556(91)  4134(85) 

NA 

Dual 

4513  (106)  4064(120) 

4668  (174) 

*Note.  Estimate  of  standard  error  in  parentheses. 


To  assess  whether  the  differences  between  conditions  were  statistically  reliable,  a  model 
predicting  log  (accuracy)  based  on  Condition  (single-dual),  Relevance,  a  Condition  x  Relevance 
interaction,  and  participant  enabled  within-subject  comparisons.  We  analyzed  log  (accuracy) 
because  residuals  from  the  raw  accuracy  scores  were  skewed  and  a  normal  distribution  better 
described  the  residuals  from  a  log  (accuracy)  model  as  assumed  by  the  linear  regression.  This 
procedure  excluded  the  trials  in  which  the  target  appeared  as  the  three  red  monitor  lights  (as  they 
were  neither  relevant  nor  irrelevant)  although  the  other  trials  from  the  dual  task  condition  were 
included.  Results  of  an  Analysis  of  Variance  (ANOVA)  on  this  model  indicated  a  statistically 
significant  effect  of  threat  relevance,  7^(1 ,45)  =  92 ,p  <  .001,  tf  =  .67,  and  no  reliable  effect  of 
either  single  versus  dual  task  condition,  A(l,45)  =  2.1  ,p  =  .15,  i /  =  .044,  nor  was  there  a  reliable 
interaction,  7^(1 ,45)  =  .65,  p  =  .42,  rj2  =  .014. 

A  second  perfonnance  measure  analyzed  was  Soldier  time-to-respond  when  they  detected 
a  target.  Similar  to  accuracy,  Soldiers  responded  faster  to  targets  in  relevant  locations  than  to 
targets  in  irrelevant  locations.  Because  response  time  was  positively  skewed,  we  performed  an 
ANOVA  on  log  (reaction  time)  which  again  revealed  a  reliable  effect  of  relevance  on  response 
time,  F( '  1 ,44)  =  6.3,  p  =.016,  rj2=  .12,  a  reliable  effect  of  the  single  and  dual  task  conditions, 
A(l,44)  =  23.7, p  <.001,  >7 2  =.345,  and  no  reliable  interaction,  7^(1 ,43)  =  .04 ,p  =  .82,  rj  2=  .001. 

Across  the  two  primary  measures  (accuracy  and  time-to-respond),  there  were  differences 
between  threat-relevant  and  threat-irrelevant  target  locations  which  is  promising  because  it 
suggests  that  measuring  such  differences  in  a  future  training  system  may  help  provide  metrics 
about  skill  level  that  can  be  used  for  assessment  and/or  feedback.  The  faster  response  time  in  the 
dual-task  condition  compared  to  single-task  condition  is  counterintuitive.  This  finding  may  be  a 
practice  effect  or  may  stem  from  an  overall  higher  level  of  arousal  induced  by  the  more  difficult 
dual-task  situation.  However,  the  reliable  impact  of  threat  relevance  indicates  that  Soldiers  were 
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typically  able  to  infer  threat-relevant  locations  and  focus  their  attention  in  those  locations  which 
degraded  their  perfonnance  on  threat-irrelevant  trials. 

To  assess  whether  Soldiers  with  more  experience  differed  in  their  ability  to  attend  to 
threat-relevant  locations,  a  threat  bias  score  was  calculated  for  each  Soldier  by  dividing  their 
mean  response  time  by  their  mean  accuracy.  The  threat  bias  score  assessed  the  extent  to  which 
detection  is  biased  toward  threat-relevant  regions  of  visual  space.  A  ratio  value  of  1.0  would 
indicate  equal  performance  for  both  types  of  targets,  values  less  that  1.0  would  indicate  a  bias 
toward  threat-relevant  locations  (i.e.,  they  were  identified  faster  and  more  often).  The  natural 
logarithm  of  these  values  was  used  to  perform  a  linear  regression  analysis  that  compared  the  log 
ratio  to  number  of  deployments  reported  by  Soldiers7.  A  negative  threat  bias  score  indicated  that 
Soldiers  were  biased  toward  detecting  and  identifying  targets  in  threat-relevant  regions.  Such  a 
bias  is  likely  to  be  adaptive  and  indicates  Soldiers  are  using  their  limited  attentional  resources  to 
monitor  the  most  likely  targets,  but  also  indicates  that  they  may  miss  seeing  threats  that  appear  in 
unlikely  places.  The  slope  (beta  =  -.3 1  log-RT  units  per  deployment,  t( 44)  =  -.39,  p  =  .69)  and 
the  intercept  (-.015)  were  not  significant.  This  indicates  that  all  Soldiers  had  a  tendency  to 
identify  targets  in  threat-relevant  locations  regardless  of  the  number  of  deployments  (Figure  6). 

Threat  bias  by  experience 
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Figure  6.  Relationship  between  bias  to  detect  threat-relevant  vs.  threat-irrelevant  locations  and 
number  of  times  deployed. 

Change  Detection.  To  assess  change  detection  ability,  each  base  image  was  coded 
according  to  its  critical  change  locations.  Coding  of  the  changes  in  each  image  indicated  the 


7  The  natural  log  transformation  was  used  to  correct  for  truncated  range  between  0  and  1  in  cases  where  mean 
accuracy  exceeded  mean  response  time. 
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presence,  absence,  or  the  change  of  a  feature.  In  addition,  coding  differentiated  each  change  as 
threat  relevant  or  threat  irrelevant.  The  image  set  contained  62  coded  changes  and  although  each 
Soldier  saw  all  62  images,  they  responded  to  only  half  the  images  totaling  3 1  changes  per 
Soldier.  This  coding  revealed  27  (44%)  changes  as  irrelevant  and  35  changes  (56%)  as  relevant. 

Results  showed  several  important  trends.  Across  all  Soldiers,  they  annotated  355  targets 
but  281,  or  79%  of  these  targets  did  not  involve  an  actual  change  between  the  two  images.  These 
281  false  alarms  were  not  coded  as  threat  relevant  or  irrelevant.  Of  the  remaining  74  identified 
targets,  7 1  were  in  threat-relevant  locations,  providing  evidence  that  participants  noticed  changes 
related  to  threats  (Table  8).  Furthermore,  participants  were  more  likely  to  identify  a  target  when 
a  feature  was  absent  in  the  first  viewing  but  present  in  the  second  viewing  (56  trials)  versus  when 
the  target  was  present  in  the  first  viewing,  but  absent  in  the  second  (15  trials). 

Table  8 

Distribution  of  target  identifications  in  change  detection  exercise 


Threat  status 

Uncoded  Irrelevant  Relevant 


Feature  present 

0 

3 

53 

Feature  absent 

0 

0 

15 

Feature  change 

0 

0 

3 

No  change 

281 

0 

0 

The  overall  high  proportion  of  false  alarms  could  be  evidence  that  change  detection  skill 
was  poor  and  may  even  indicate  random  guessing.  However,  the  proportion  of  targets  actually 
found  when  Soldiers  had  no  warning  they  should  look  for  changes,  indicates  an  existing  ability 
to  detect  change.  Although  only  about  20%  of  the  responses  were  for  actual  changes  Soldiers 
selected  about  3%  of  the  image  areas  as  the  change  zones  (ranging  from  0.37%  to  12.7%  across 
images).  Thus,  Soldiers  were  about  6.7  times  more  likely  to  indicate  actual  changes  than  if  they 
had  been  responding  by  chance.  To  assess  whether  performance  differed  reliably  from  chance, 
mean  accuracy  rates  (the  number  change  targets  identified  relative  to  the  total  number  of 
attempts)  were  compared  to  the  proportion  expected  by  chance.  Forty-six  Soldiers  took  part  in 
the  exercise,  but  two  provided  no  responses.  Across  the  remaining  44  Soldiers,  the  mean 
proportion  of  attempts  that  identified  a  change  target  was  .233  with  a  99%  confidence  interval 
between  .14  and  .32  (three  Soldiers  did  not  complete  the  change  exercise  because  of  time 
restrictions).  The  chance  proportion  of  .03  identified  change  targets  lies  far  outside  the  region, 
*(42)  =  5.9, p  <  .0001,  Cohen’s  d  =  0.90. 

Because  we  were  interested  in  how  change  detection  occurs  in  an  applied  setting  (as 
opposed  to  in  a  laboratory),  we  made  use  of  what  is  commonly  referred  to  as  the  incidental 
approach  (Levin  &  Simons,  1997).  In  these  situations,  participants  are  not  informed  that  a 
change  or  difference  will  occur  and  they  are  often  asked  to  perform  an  intermediary  task  as  their 
primary  task.  In  Rensink’s  review  (2002,  p.  257)  he  notes,  “The  degree  of  [change]  blindness 
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encountered  is  generally  much  higher  than  that  found  using  intentional  approaches.  However, 
some  ability  to  detect  change  remains.”  The  results  of  the  change  detection  exercise  used  in  this 
research  support  that  idea.  Based  on  the  results,  Soldiers  may  be  considered  to  have  had  a  lower 
response  threshold  or  a  high  false  alann  rate.  However,  when  they  indicated  changes  that  were 
indeed  present,  they  were  most  likely  at  threat-relevant  locations.  Given  the  severe 
consequences  of  not  detecting  threat-relevant  changes  in  the  operational  environment,  adopting  a 
lower  threshold  is  an  adaptive  strategy. 

Interviews:  Threat  Detection  Model. 

The  analysis  presented  in  this  section  aimed  to  link  the  comments  from  Soldiers  with 
deployment  experience  to  the  Threat  Detection  Model  proposed  by  Zimmerman  et  ah,  (In 
Preparation).  These  comments  provide  select  examples  of  behaviors  that  map  onto  the  loop  (see 
Figure  1).  The  next  section  provided  examples  specifically  relevant  to  expertise  development. 

Experienced  Soldiers  often  discussed  threat  detection  exercises  that  related  to  their 
cognitive  and  perceptual  processes  such  as  encoding  details  about  the  situation  (see  Threat 
Detection  Loop,  Figure  1,  box  a).  For  example,  Soldiers  made  the  following  comments: 

•  “Just  keep  an  eye  out. .  .to  keep  an  eye  out  for  a  vehicle  and  right  in  here  where  they’re 
sitting.” 

•  “You  have  to  focus  on  everything,  pay  attention  to  what  they’re  wearing.” 

•  “So,  I  just  watched  people. .  .1  know,  okay  this  time  of  day,  these  people  are  going  out  and 
this  time  of  day,  they’re  coming  back.” 

Soldiers  also  made  comments  about  their  infonnation  gathering  processes  that 
demonstrate  their  need  to  balance  making  decisions  about  a  threat  with  the  need  to  gather 
infonnation  (f). 

•  “It  should  be  with  the  local  people;  they  (will)  give  away  everything. .  .they’ll  come  and 
tell  you.” 

•  “Presence  patrols;  we’d  do  a  lot  of  them.  And  a  lot  of  times  we’d  go  to  each  shop  and 
talk  to  people  in  there  and  ask  them  if  they  noticed  anything  weird  in  the  area.” 

Soldiers  with  experience  often  described  the  challenges  related  to  properly  classifying 
threat  cues  and  threat  inelevant  cues  (b)  and  their  ability  to  detect  changes  in  an  environment  (g). 
They  reported: 

•  “It’s  dirt  and  gravel  which  is  not  cool  because  it’s  too  easy  to  mess  with  dirt  and  gravel 
and  make  it  look  the  same.  Pavement  is  a  little  bit  easier  to  notice  differences.” 

•  “I  really  don’t  like  looking  for  IEDs:  too  many  places  to  put  it,  to  hide  it,  hard  to  find,  not 
enough  manpower.” 

•  “Really,  no  matter  how  you  looked  at  it,  you  could  look  at  almost  anyone  and  decide  if 
you  think  they’re  suspicious.” 
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•  “I  mean  they  changed  their  tactics  every  day  and  every  time  we  changed  ours,  they’d 
change  theirs.  So  it’s  hard  to. .  .one  day  it  could  be  a  coke  can  sitting  right  there  and  the 
next  day  it  could  be  a  tire.  You  just  never  know.” 

Soldiers  also  indicated  that  they  often  relied  on  prior  experience  and  knowledge  to 
detennine  their  search  strategies  and  this,  in  turn,  influenced  their  ability  to  detect  threats.  An 
example  statement  provides  illustration  of  incorporating  prior  experiences  into  decisions  (d): 

•  “You’ll  see  a  lot  of  the  areas  that  they’ve  used  before.  And  it’s  a  big  give  away,  like 
they’ll  use  telephone  poles  as  markers  and  they’ll  use  light  poles,  they’ll  use  trees, 
intersections.” 

Interviews:  Expertise  Development. 

This  section  provides  findings  from  the  interviews  that  represent  typical  responses  from 
Soldiers  who  have  never  deployed,  those  who  have  deployed  once,  and  those  who  have  deployed 
two  to  four  times.  The  purpose  of  this  analysis  was  to  identify  differences  in  causal  reasoning 
skills  between  less  and  more  experienced  Soldiers  as  they  identified  and  discussed  possible 
threats  in  photographs  depicting  real-world  situations.  The  results  contain  infonnation  given  by 
Soldiers  in  response  to  the  photographs  that  pertain  to  the  high-level  categories  (Table  9). 
Soldiers  in  all  three  groups  (0,  1,  or  2-4  deployments)  were  able  to  identify  or  discuss  types  of 
threats,  threat  cues,  and  threat  detection  exercises.  Soldiers  with  deployment  experience  more 
often  discussed  strategies  for  threat  detection,  threat  detection  skills,  challenges  in  detecting 
threats,  and  solutions.  The  interview  analyses  explored  the  content  of  these  differences  by 
comparing  comments  made  by  less  and  more  experienced  Soldiers  in  each  category. 

Table  9 

Number  of  Soldiers  who  made  comments  in  each  interview  data  category 


Never 

Types 

of 

Threats 

Threat 

Cues 

Strategies 

for 

Threat 

Detection 

Threat 

Detection 

Tasks 

Threat 

Detection 

Skills 

Challenges 

in 

Detecting 

Threats 

Solutions 

Deployed 
*N  =  13 

1 

Deployment 
N=  14 

2-4 

13 

13 

8 

8 

2 

4 

0 

15 

15 

14 

7 

7 

14 

4 

Deployments 
N=  19 

15 

14 

13 

8 

6 

10 

2 

*Note:  N  =  Number  of  participants  who  made  a  comment  in  each  section. 
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Participants  provided  overall  assessments  of  the  photos  they  viewed.  To  exemplify  these 
discussions  two  Soldiers,  a  junior  enlisted  Soldier  who  had  never  deployed  and  an  experienced 
Soldier  who  had  deployed  once,  directed  their  comments  on  a  photograph  taken  of  a  cemetery 
(Figure  7).  The  photo  they  discussed  depicted  three  members  of  a  U.S.  force  in  a  cohesive  group 
and  one  member  who  is  by  himself  as  they  are  walking  among  numerous  headstones.  During  the 
interviews,  both  Soldiers  suggested  that  perhaps  the  U.S.  forces  were  pursuing  someone  through 
the  cemetery  or  trying  to  clear  the  area.  The  experienced  Soldier  elaborated  on  this  suggestion, 
“or  they’re  just  looking  through  to  clear  it  and  maybe  make  sure  there  are  no  explosives  and 
nobody  hiding  in  there”  and  considered  additional  threats,  specifically  from  snipers. 


Figure  7.  Photo  discussed  by  Soldiers  that  revealed  experienced-based  reasoning  differences. 


The  junior  enlisted  Soldier  focused  his  comments  on  the  individual  separated  from  the 
group  and  the  threat  to  the  lone  Soldier  stating  “That  guy  by  himself  is  not  a  very  smart  idea. 
They  (the  enemy)  could  be  all  up  over  here. .  .basically,  right  here,  this  is  not  right  when  you’re 
clearing  anything,  you  stay  together;  you  don’t  ever  separate.”  This  demonstrated  the  more 
procedural  and  context-free  rules  to  thinking  often  exhibited  by  novices  (Dreyfus  &  Dreyfus, 
1986).  The  experienced  Soldier  was  also  concerned  with  the  Soldier  separated  from  the  group, 
but  provided  suggestions  for  why  he  separated  from  the  others: 

[T]his  guy  is  on  his  own  and  maybe  instead  of  going  on  his  own,  he  ought  to  go  with 
them.  Obviously,  these  guys  are  clearing  around  this  area...  they  may  need  somebody  up 
there,  that  way  if  something  does  happen  they  have  the  high  ground  already  so  [that  the 
lone  Soldier]  could  see  around  and  watch  for  anything  coming  up  on  them. 
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This  Soldier  was  demonstrating  his  ability  to  create  a  causal  story  about  a  threat  situation  (Figure 
1,  box  h)  and  to  hypothesize  about  possible  explanations  for  why  the  solitary  Soldier  was 
breaking  from  procedure  and  taking  what  the  junior  enlisted  Soldier  perceived  as  unsafe  action, 
demonstrating  a  more  proficient  skill  level. 

The  junior  enlisted  Soldier  focused  on  procedural  rules  discussing  how  Soldiers  should 
not  enter  the  cemetery  unless  they  have  received  permission.  The  experienced  Soldier  thought 
beyond  the  situational  constraints  and  discussed  why  this  break  in  procedure  might  be  a  good 
strategy,  “I’d  make  sure  that  we  have  high  ground  in  case  there’s  anybody  hiding  in  there  as 
we’re  going  through  it,  to  watch  for  somebody  running  around...”  This  difference  illustrated 
how  decision  makers  incorporate  their  experience  into  their  assessments  of  situations  (d)  to 
interpret  what  they  are  seeing  (Dreyfus  &  Dreyfus,  1986,  Ross,  et  al.,  2005).  The  experienced 
Soldier’s  comment  also  demonstrated  that  he  was  able  to  look  at  high-priority  threat  locations 
based  on  knowledge  (i)  by  taking  action  that  reduces  the  threat. 

Types  of  Threats. 

Table  10  lists  the  types  of  threats  mentioned  by  Soldiers  when  viewing  the  photos  and  the 
number  of  Soldiers  who  made  explicit  reference  to  these  threats.  Soldiers  identified  these  threats 
spontaneously  rather  than  from  a  list  of  threat  types,  thus  similar  types  are  listed  separately.  For 
instance,  some  Soldiers  described  bombs  and  explosives  while  others  made  more  specific 
reference  to  IEDs  and  rocket-propelled  grenades  (RPG),  etc.  While  Soldiers  in  each  deployment 
category  discussed  IEDs  most  often,  followed  by  snipers,  some  differences  appeared  between 
those  who  had  deployed  compared  to  those  who  had  not  deployed.  More  Soldiers  in  the 
deployment  groups  referenced  IEDs  compared  to  those  who  had  never  deployed.  Soldiers  who 
had  never  deployed  and  those  who  had  experienced  between  two  and  four  deployments  both 
referenced  10  threats  while  Soldiers  who  had  experienced  one  deployment  refer  to  13  types  of 
threats.  This  may  be  because  non-deployed  experienced  Soldiers  relied  on  threats  they  learned 
about  in  training  or  might  expect  to  encounter  in  a  particular  situation.  Soldiers  who  have 
deployed  once  have  a  larger  list  of  possible  threats.  This  may  simply  be  due  to  changing  enemy 
TTPs  or  may  be  because  they  have  started  to  expand  their  knowledge  of  possible  threats,  but 
have  not  yet  narrowed  down  the  list  of  potential  threats  to  those  that  are  the  most  likely.  Perhaps 
the  most  experienced  Soldiers  have  narrowed  down  their  list  of  threats  because  they  are  better 
able  to  filter  and  sort  threats  according  to  relevance  and  likelihood  and  may  indicate  they  have 
more  developed  schemas  and  a  more  proficient  skill  level  (Dreyfus  &  Dreyfus,  1986). 

Soldiers  could  have  discussed  any  possible  types  of  threats  during  their  interview. 

Specific  enemy  and  troop  tactics  such  as  those  discussed  in  the  cemetery  scene  (Figure  7)  are  just 
one  example.  Other  differences  between  less  and  more  experienced  Soldiers  concerned  their 
discussion  of  threats  posed  by  the  environment.  Less  experienced  Soldiers  often  mentioned  the 
terrain.  One  Soldier  stated,  “Just  the  fact  that  they’re  around  the  mountains;  the  visibility  around 
those  mountains  is  limited.  The  terrain  -  it’s  easy  to  blend  in  and  get  assaulted  or  ambushed 
from  any  angle,  from  the  top.”  Another  typical  comment  indicated  the  uncertainty  of  the 
situation,  as  one  Soldier  stated  “High  elevation;  you’re  at  the  bottom  of  the  creek  bed  and  people 
could  be  above  you.  You  really  don’t  know  what’s  up  here.  You  might  have  scouts  going  every 
way  but,  right  here,  you  look  like  you’re  by  yourself.” 
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Table  10 


Number  of  Soldiers  who  discussed  each  common  type  of  threat * 


Threat  Type 

Never  deployed 
(/i  =  13) 

Soldier  Experience 

1  Deployment 
(n  =  14) 

2-4  Deployments 

(/i  =  20) 

IED 

5  (38%) 

10(71%) 

10  (50%) 

VBIED 

3  (23%) 

4  (29%) 

0  (0%) 

PBIED 

1  (8%) 

3  (21%) 

2(10%) 

Bomb/Explosives 

4(31%) 

1  (7%) 

6  (30%) 

Sniper 

4(31%) 

6  (43%) 

4  (20%) 

RPGs 

1  (8%) 

2  (14%) 

2(10%) 

EFPs 

0  (0%) 

2  (14%) 

1  (5%) 

Ambush 

2  (15%) 

1  (7%) 

1  (5%) 

Small  Arms 

0  (0%) 

2  (14%) 

3  (15%) 

Grenade 

0  (0%) 

2  (14%) 

0  (0%) 

Mortar 

0  (0%) 

2  (14%) 

0  (0%) 

Surveillance 

2  (15%) 

3  (21%) 

0  (0%) 

Other 

1  (8%) 

6  (43%) 

1  (5%) 

*Note.  Soldiers  could  discuss  more  than  one  type  of  threat. 


Three  Soldiers  who  had  deployed  once  made  comments  about  the  terrain;  one  stated,  “A 
curve  in  the  road  was  the  worst.  That’s  where  they  seem  to  hit  you  the  most  because  they  could 
see  your  vehicle  turn  from  a  long  ways.”  This  Soldier’s  comment  shows  that  he  was  able  to 
identify  high-priority  threat  locations  based  on  his  previous  knowledge  (i).  Another  Soldier 
stated  that  snipers  were  a  threat  and  “it’s  a  linear  danger;  it’s  an  open  space  and  there’s  more 
cover  outside  than  where  you’re  at.  It  could  be  a  vast  amount  of  threats;  there  are  a  million 
different  scenarios.”  The  final  Soldier  said  “Maybe  it  is  a  high  ground  so  if  there’s  a  road  here  to 
make  sure  there’s  nobody  hiding  up  there  -  maybe  a  sniper  or  something  like  that.”  These 
Soldiers’  comments  indicated  that  they  were  actively  trying  to  identify  the  threat  and  create 
stories  about  the  threat  situations  (h)  while  still  operating  with  a  good  deal  of  uncertainty  as  they 
tried  to  determine  what  action  they  would  take. 

Finally,  two  Soldiers  who  had  deployed  multiple  times  commented  on  the  terrain.  One 
Solider  made  a  statement  about  how  open  spaces  make  troops  vulnerable  to  sniper  fire.  The 
other  created  a  story  to  explain  how  he  thought  terrain  was  the  biggest  threat.  He  stated: 

In  my  opinion ,  terrain  plays  the  biggest  role  in  everything.  It  pretty  much  decides 
everything,  whether  there ’s  a  bunch  of  kids  and  maybe  the  particular  enemy  in  that  area 
doesn ’t  like  them  or  doesn ’t  like  to  take  out  children.  IED  spots  and  sniper  spots,  any 
advantage  point  in  an  ambush  and  things  of  that  sort. 

He  then  explained  how  he  might  take  advantage  of  the  terrain  if  he  were  the  enemy.  “I’d  just 
keep  to  the  right  side  right  there  and  try  to  take  out  part  of  the  road;  you  could  also  put  one  in  the 
mountain  right  here.  That’s  not  a  lot  of  damage  to  the  truck  but,  again,  I’d  pile  a  bunch  of  dirt 
and  rock  on  the  road  and  now  you  have  vehicles  trapped  on  either  side.”  This  Soldier 
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demonstrated  ability  to  take  the  enemy’s  perspective  to  reason  about  possible  threats  and 
consequences  of  enemy  actions,  indicating  a  higher  level  of  causal  reasoning. 

Threat  Cues. 

Less  experienced  Soldiers  tended  to  discuss  specific  cues  and  make  general  comments 
about  behavior  such  as  “they  look  kind  of  suspicious”  or  “being  by  himself,  doesn’t  look  right.” 
These  Soldiers  tended  to  describe  suspicious  behaviors  in  general  terms,  for  instance,  “I  guess 
the  biggest  thing  would  be  any  specific  individuals  who  brought  attention  to  themselves,  just 
through  their  actions,  just  wherever  I’m  observing  at  the  moment.”  These  Soldiers  also 
mentioned  environmental  cues,  specifically  rock  piles,  garbage  piles,  buildings  with  windows, 
tinted  windows,  lowered  vehicles,  and  wires. 

Soldiers  with  one  or  more  deployments  typically  listed  threat  cues,  but  also  provided 
more  contextual  details  and  interpretations  of  the  cues  and  situations.  Some  of  the  cues 
discussed  were: 

•  someone  sitting  in  a  car  which  might  indicate  they  have  a  detonation  device, 

•  crowds  of  people, 

•  people  who  act  suspicious  such  as  looking  at  you  then  running  off  to  hide  or  trying  to 
stay  hidden, 

•  piles  of  junk  or  garbage, 

•  animals  carrying  heavy  loads, 

•  people  with  bulky  or  odd  clothing, 

•  anything  that  seems  out  of  place, 

•  lights  that  have  gone  off, 

•  telephone  poles  as  markers,  spotters, 

•  damaged  road  medians  or  discolored  curb, 

•  wires, 

•  locals  avoiding  certain  areas,  and 

•  police  not  doing  their  jobs. 

Many  Soldiers  in  this  group  also  mentioned  that  any  changes  they  were  able  to  detect 
might  be  a  threat  cue.  For  example,  one  Soldier  described  a  threat  “If  you’ve  been  down  this 
route  so  many  times  and  it’s  always  been  the  same  and  then  all  of  a  sudden  there’s  a  car  there 
just  dead.”  These  comments  indicate  that  these  Soldiers  are  aware  that  the  ability  to  detect 
change  or  anomalies  in  the  environment  (g)  is  an  important  skill. 

These  experienced  Soldiers  also  reasoned  about  why  specific  cues  may  or  may  not 
indicate  a  threat.  For  example,  one  Soldier  stated  “There  is  a  possibility  of  an  IED  somewhere 
around  here,  but  it’s  not  likely  because  this  is  a  big  puddle.  It  looks  like  it  just  recently  rained  so 
they’re  probably  not  going  to  put  their  electronic  stuff  out  because  it’d  get  ruined  so  they  can’t 
really  use  IEDs.”  A  few  Soldiers  discussed  why  lights  might  be  a  cue  such  as  “Lights. . . 
sometimes  people  notice  the  lights  go  off  when  there  are  things  going  to  happen...  Sometimes  the 
lights  will  be  out  and  sometimes  they  will  just  go  off  too.  So  it’s  all  variable.”  These  Soldiers 
were  able  to  incorporate  prior  infonnation  into  their  decisions  (d)  and  create  causal  stories  about 
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threat  situations  (h)  by  discussing  their  experiences  with  light  cues  and  the  likelihood  that  they 
are  cues  to  threat.  This  suggests  they  are  able  to  classify  cues  as  threats  or  non-threats  (b)  based 
on  the  context  of  the  situation. 

Strategies  for  Threat  Detection. 

Many  of  the  less  experienced  Soldiers  made  comments  about  spotting  cues  as  a  strategy 
for  threat  detection.  They  made  comments  about  maintaining  alertness,  looking  for  triggermen 
or  spotters,  and  other  non-specific  comments  such  as  “There  are  just  different  areas  that  you’d 
check  regularly.”  One  less  experienced  Soldier  constructed  a  more  detailed  narrative  about  a 
situation  depicted  in  Figure  8  and  stated  that  “If  we  came  up  here  we’d  have  the  guy  step  away 
from  the  wagon  and  search  the  people  on  the  wagon.  Have  people  pull  security  while  you  were 
doing  it.  Check  the  cart,  check  their  bags,  and  you’d  go  over  and  check  the  motorcycle  to  see  if 
anything  looks  out  of  place.  Go  into  the  house;  clear  the  house.  I  guess  just  talk  to  the  people 
and  see  if  they  know  any  of  them.”  He  continued  to  explain  “We’d  walk  by  a  vehicle,  look 
around  at  the  people  and  find  out  what  they’re  doing.  If  they’re  moving  away,  something  might 
be  up.  Find  out  if  it’s  a  suspicious  vehicle,  deploy  your  security  to  look  inside  and  make  sure  it’s 
not  trapped,  open  up  the  doors,  take  a  look  inside  the  glove  compartment,  the  console, 
underneath  the  seats,  and  dash.  You’d  want  to  pop  the  hood  and  trunk  to  look  in  there.  Look  for 
any  wires  that  shouldn’t  be  there.”  Finally,  this  Soldier  suggested  he  would  ask  locals  about  any 
possible  threats  in  the  area.  His  analysis  indicates  that  perhaps  he  is  moving  from  the  novice 
skill  set  to  that  of  advanced  beginner  or  competent  stages  because  he  is  processing  the 
information  and  searching  for  plausible  solutions  within  the  context  of  the  situation  (Dreyfus  & 
Dreyfus,  1986). 


Figure  8.  Photo  that  led  to  discussion  with  Soldiers  concerning  threat  strategies. 
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Another  strategy  recommended  by  the  Soldiers  who  had  one  or  more  deployments  was 
the  use  of  dismounted  patrol.  One  Soldier  stated  he  would  “send  your  dismounts  out  to  find 
them,  to  look  for  wires,  and  to  look  for  stuff’  while  another  mentioned  that  dismounted  patrols 
allowed  for  the  detection  of  a  command  wire.  He  said  “I  actually  tripped  on  a  command  wire 
one  day  and  that’s  how  we  found  it.  Luckily,  we  did  that  dismounted,  walked  around  and  looked 
for  command  wire.”  Many  Soldiers  commented  that  they  would  look  for  command  wire  as  a 
way  to  detect  IEDs.  This  statement  also  echoed  the  realization  that  often  no  cues  were  present 
until  the  IED  detonated.  Another  mentioned  “. .  .1  can’t  say  that  we  looked  for  (cues  to  IEDs) 
because  we  never  actually  found  an  IED  before  we  detonated  it.  Either  it  just  went  off  on  us  or  a 
patroller  has  already  found  one  and  had  already  blocked  traffic.” 

Soldiers  who  had  deployed  once  were  interested  in  using  the  local  population  as  a 
strategy  for  threat  detection.  As  one  Soldier  put  it,  “[Threat  detection]  should  be  with  the  local 
people;  they  give  away  everything.  And  usually  in  populated  areas  there’ll  be  no  threat.  I’ve 
never  had  an  issue  with  the  population. .  .they’ll  come  and  tell  you  because  they  don’t  want 
[terrorist  activity]  there  just  as  much  as  you  don’t  want  it  there.”  Leveraging  the  local 
populations  was  a  common  theme  among  many  of  the  Soldiers  who  had  deployed.  One  Soldier 
with  multiple  deployments  relied  on  an  interpreter  to  detect  threats.  He  stated  that  “if  you  have  a 
good  interpreter  he’ll  say  ‘he’s  speaking  a  really  odd  form  of  Arabic  or  he  doesn’t  sound  like  a 
local.’  So  a  lot  of  the  interpreters  are  local,  not  from  the  city  but  from  the  area,  so  it’d  be  like  a 
southerner  saying  he  speaks  like  a  northerner  or  western. . .  [information  like  that]  just  makes  sure 
you  look  a  little  closer.” 

With  the  exception  of  the  one  non-deployed  Soldier  quoted  above,  most  Soldiers  who  had 
not  deployed  were  unable  to  provide  detailed  assessments  of  strategies  to  detect  threats.  Only  8 
of  13  Soldiers  identified  strategies.  Those  who  did  tended  to  mention  non-specific  strategies. 
Soldiers  who  had  deployment  experience  often  gave  richer  narratives  of  what  strategies  they 
would  take.  Additionally,  Soldiers  who  had  never  deployed  did  not  address  topics  such  as  the 
lack  of  strategies,  using  local  populations,  or  dismounted  patrol  as  strategies  for  threat  detection. 

Threat  Detection  Tasks. 

Eight  non-deployed  Soldiers  stated  that  observing  situations  and  environments  was  a 
main  threat  detection  task.  They  made  comments  about  checking  the  area,  people,  or  objects. 
One  Soldier  said  “Hopefully,  I’d  be  with  other  people  and  I’d  say  ‘keep  an  eye  on  them.’  And 
get  other  people  looking  around  the  perimeter,  a  360.”  Another  Soldier  said  “I’d  just  pay 
attention  to  the  people,  the  vehicles  around  the  buildings,  to  see  in  the  windows,  or  even  the  top 
of  the  buildings  because  people  get  up  there  and  stuff.  And  I’d  pay  attention  to  that  a  lot.” 
Another  Soldier  commented  “It’s  just  that  you  have  to  check  people;  that  is  what  we  were  told 
[in  training].  If  the  people  are  in  your  area  and  they  could  potentially  be  a  threat  then  you  should 
check  them.”  These  less  experienced  Soldiers  largely  described  the  tasks  from  a  passive  point  of 
view  and  focused  primarily  on  tasks  reinforced  through  training,  as  the  last  Soldier  indicated. 

While  less  experienced  Soldiers  focused  on  scanning  and  observing  the  environment,  the 
Soldiers  who  had  deployed  multiple  times  often  took  a  more  active  stance  in  their  threat 
detection  tasks.  One  Soldier  described  searching  someone  riding  a  motorcycle  and  said  “We’d 
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stop  them  and  maintain  standoff,  have  him  park  his  motorcycle,  get  off,  and  lift  his  arms  up  into 
a  circle.  Try  and  determine  if  he  was  a  threat  or  not. .  .we’d  just  tell  him  stop  and  then  we’d  have 
him  open  his  hands  and  make  sure  he  didn’t  have  any  trigger  mechanisms  or  anything  like  that  in 
his  hands  or  running  through  his  sleeve.  We’d  see  if  the  rest  of  his  coat  fell  down,  like  when  he 
stood  up  see  if  his  coat  fell  down  as  opposed  to  staying  up,  bulking,  you  know.”  This  Soldier 
created  stories  about  the  threat  situations  (h)  and  incorporated  prior  knowledge  to  hypothesize 
about  what  might  happen  and  what  actions  he  might  take  (d).  This  Soldier  mentally  simulated  a 
situation  in  which  the  individual  posed  a  threat  (h)  “We’d  continue  standoff  and  then  go  check  it 
out. .  .I’d  look  at  it  with  optics  this  way,  and  then  I’d  make  a  big  circle  around  over  here,  and  then 
I’d  look  at  it  with  optics  this  way,  and  I’d  get  that  dude  on  his  face.  No  chance  he’s  going  to 
blow  me  up  while  he’s  laying  on  his  face.”  This  comment  also  demonstrated  ability  to  take 
action  to  the  better  classify  the  threat  (b)  and  take  action  to  reduce  the  threat  (e).  He  continued  to 
suggest  what  might  happen  if  they  were  attacked  and  constructed  a  rich  narrative  that  included 
checking  for  a  secondary  device,  establishing  security,  using  dismounts  to  search  for  command 
wire,  and  maintaining  visual  contact  and  communication.  His  ability  to  construct  a  rich 
description  demonstrates  many  aspects  of  the  threat  detection  loop  including  being  able  to 
encode  details  about  the  situation  (a),  detect  changes  or  anomalies  (g),  balance  his  decision  with 
the  need  to  gather  additional  infonnation  (f),  and  identify  the  appropriate  course  of  action  (e). 

Some  Soldiers  with  one  deployment  focused  on  tasks  that  involved  understanding  the 
overall  context  of  the  threat.  One  Soldier  described  presence  patrols  as  a  task  where  “you’re  just 
looking  around,  checking  for  things  that  could  be  bad,  meeting  people  and  you’ll  be  meeting 
high-ups,  mean  low-down  people,  everything  -  just  making  your  presence  known.”  Another 
Soldier  thought  that  one  of  the  most  important  tasks  was  to  interact  with  locals  to  understand  the 
broader  situation.  He  stated  “The  biggest  thing  would  be  to  know  all  of  the  rumors  going  into  a 
situation;  that  this  guy  may  or  may  not  know  that  we  know  about  him.  Essentially,  whether  he 
has  ties  to  foreign  fighters,  al  Qaeda,  Taliban,  or  whether  he’s  like  one  of  the  most  trustworthy, 
most  loyal  American  supporters  that  the  country  has.”  Another  Soldier  described  the  difference 
in  detecting  threats  between  conducting  a  presence  patrol  rather  than  patrol  based  on  a  tip.  He 
said  that  during  presence  patrols  “You  still  look  for  threats,  but  you’re  really  going  to  treat  the 
Iraqis,  you’d  talk  to  them,  just  like  a  regular  person.  You  wouldn’t  search  them,  it’s  more  of  a 
trusting  type  thing.”  Soldiers  who  deployed  once  focused  on  how  the  tasks  fit  into  the  broader 
context  of  the  situation  and  explained  that  knowledge  of  the  local  allegiances  detennined  how 
they  went  about  threat  detection.  Often,  the  Soldiers  with  no  deployment  experience  focused  on 
the  procedures  they  would  engage  in  to  stay  safe  without  taking  into  consideration  the  large 
goals  of  community  relationships  and  intelligence  gathering. 

Threat  Detection  Skills, 

Only  two  non-deployed  Soldiers  commented  on  skills  related  to  threat  detection.  One 
mentioned  that  he  relied  on  life  experience  and  the  threats  encountered  in  everyday  life  to 
detennine  threats  while  the  other  engaged  in  thinking  like  the  enemy.  He  suggested  that  it  would 
be  easy  to  hide  something  in  a  sack  of  papers  that  a  donkey  was  carrying.  He  said  “Yes,  it  seems 
like  it’d  be  real  easy  to  do  that,  anyway,  if  I  was  going  to  hide  something  it  seems  like  right  at 
the  bottom  of  one  of  those  sacks  would  be  perfect.”  He  mentioned  that  seeing  a  scene  like  this 
would  raise  his  suspicion  level  because  “it  just  seems  like  it  provides  (the  donkey  owner)  an 
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opportunity  to  get  close  without  actually  having  to  go  through  security  points  or  anything  like 
that.”  While  the  Soldier  was  thinking  of  places  the  enemy  might  place  an  IED,  he  focused  on  the 
ease  of  emplacement  and  the  chance  that  the  situation  might  go  unnoticed.  These  statements 
indicated  that  this  Soldier  is  moving  to  advanced  beginner  stages  of  skill  development  as  he 
incorporated  situational  elements  into  his  threat  assessment  (Dreyfus  &  Dreyfus,  1986). 

Exercises  in  thinking  like  the  enemy  promote  the  development  of  these  skills  and  assist  Soldiers 
to  advance  beyond  novice  stages  of  skill  development. 

One  Soldier  who  had  deployed  multiple  times  took  a  different  perspective  when  thinking 
like  the  enemy  as  he  discussed  the  outcome  of  the  threat  (Figure  9)  rather  than  focusing  on 
emplacing  an  IED.  “They  have  vehicles  and  why  not  plant  an  IED  right  here;  even  if  it  doesn’t 
take  out  the  vehicle  it’s  going  to  take  out  the  road,  cut  the  lines  in  half  so  vehicles  can’t  get  back 
and  forth.  The  vehicle  will  roll  down  that  hill  as  easy  as  hell. .  .and  even  if  the  whole  road 
doesn’t  go  or  the  vehicle  rolls  down  then,  again,  you  still  have  a  giant  gap  and  this  (Soldier)  can’t 
cross,  that  one  can’t  cross.”  This  Soldier  demonstrated  causal  reasoning  about  the  threat 
situation  (h)  by  suggesting  that  IED  placement  could  result  in  multiple  outcomes.  Another 
Soldier  considered  enemy  strategy  stating  he  would  be  unlikely  to  drive  down  a  road  “because  of 
all  this  overturned  [dirt],  [the  enemy]  could  be. .  .trying  to  funnel  you  into  a  certain  side  of  the 
road  or  whatever.”  Both  Soldiers  were  trying  to  anticipate  the  intent  of  the  enemy  when  thinking 
about  what  tactics  the  enemy  would  employ  in  each  situation  rather  than  solely  focusing  on  how 
one  might  emplace  an  IED. 


Figure  9.  Photo  that  led  to  discussion  with  Soldiers  concerning  thinking  like  the  enemy  as  a 
threat  detection  skill. 
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Challenges  in  detecting  threats. 


Comments  made  by  two  of  the  four  less  experienced  Soldiers  indicated  they  had 
incomplete  mental  models.  One  Soldier  said  it  would  be  difficult  for  him  to  determine  which 
cues  to  notice  “It’s  hard  for  me  to  say  at  this  moment.  I  mean,  in  the  moment  you’d  probably  be 
able  to  identify  stuff  here  and  there,  but  talking  about  it  from  this  point  of  view  I  don’t  know 
exactly  what  I’d  be  looking  for;  just  something  that  sticks  out  from  the  rest  of  them.”  The  other 
Soldier  thought  it  would  be  difficult  to  notice  behavioral  cues  and  stated  “People  acting 
suspiciously;  I  don’t  really  know  very  much  about  that.  I  mean  I  can  sort  of  tell;  just  life 
experience,  but  if  I  could  get  some  training  on  how  to  read  people  a  little  better  it’d  be  helpful.” 
Both  statements  suggest  that  while  they  may  know  what  general  cues  to  look  for  such  as  some 
unusual  or  suspicious  behavior  they  lack  the  mental  models  to  articulate  what  these  cues  actually 
mean. 


Many  of  the  Soldiers  who  deployed  once  took  a  different  approach  to  noticing  cues. 

They  often  commented  that  most  changes  in  the  environment  were  not  indicators  of  an  attack 
and  that  because  enemy  tactics  changed  often  it  was  hard  to  distinguish  relevant  from  irrelevant 
cues.  One  Soldier  stated  “[The  enemy  is]  so  creative,  [their  tactics]  change  every  day  so  you 
can’t  really  worry  about  it.  All  you  have  to  do  is  your  job.”  Another  Soldier  stated  “Things 
change  all  the  time.  I  mean  you  could  go  down  a  route  six  times  and  have  cones  in  different 
areas  and  all  at  these  random  checkpoints  and  stuff.”  He  went  on  to  create  a  narrative  about  how 
enemies  do  their  homework  including  using  camera  operators  and  note  takers  to  watch  how  the 
U.S.  forces  respond  to  an  initial  attack  so  they  can  set  up  a  secondary  attack  as  the  quick  reaction 
force  (QRF)  or  medevac  to  arrive  on  the  scene.  This  suggests  that  these  Soldiers  have  ideas  of 
what  cues  may  indicate  attacks,  but  the  realities  of  the  situation  require  them  to  look  beyond 
specific  cues  that  may  not  be  consistently  relevant  (b).  To  do  this,  they  need  to  encode  details 
about  the  situation  (a),  engage  in  causal  reasoning  (h),  and  balance  decisions  with  infonnation 
gathering  (f). 

Soldiers  with  multiple  deployments  also  said  it  was  difficult  to  notice  cues  because  the 
enemy  changed  tactics  often  as  a  way  to  reduce  their  predictability.  One  Soldier  said  “I  really 
don’t  like  looking  for  IEDs. .  .too  many  places  to  put  it,  to  hide  it,  hard  to  find,  and  not  enough 
manpower.”  Another  Soldier  stated  that  “when  you’re  on  patrol  you’re  concerned  about 
everything...  and,  of  course  you  can’t  look  at  everything,  you’re  going  to  miss  something...” 

This  same  Soldier  went  on  to  say  “A  lot  of  the  younger  guys  who  have  never  been  [deployed], 
and  some  of  them  have  been  to  classes,  then,  yes,  the  only  thing  they’re  going  to  look  for  is 
stacked  up  rocks  and  overturned  dirt.  Yes,  one  spot  is  overturned,  they’re  going  to  look  at  the 
small  stuff,  they’re  not  going  to  look  at  the  overall  [picture] . .  .that’s  what  I’m  seeing  when  I  look 
at  this.  I’m  seeing  that  anything  can  be  a  threat.”  These  Soldiers  understand  the  role  experience 
plays  in  identifying  threat  and  non-threat  features  (b)  as  well  as  appropriately  encoding  details 
about  the  situation  (a).  These  more  experienced  Soldiers  are  able  to  focus  their  attention  on 
relevant  cues  and  reason  about  what  cues  are  not  important  or  relevant  in  the  moment.  They  are 
better  able  to  organize  and  prioritize  information  based  on  their  experience  which  research 
indicates  is  a  characteristic  of  expertise  development  (Ericsson  &  Charness,  2004). 
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Across  both  groups  of  Soldiers  who  had  deployed,  the  reliability  of  information  posed  a 
specific  challenge.  A  Soldier  who  deployed  multiple  times  gave  a  mixed  response  when  asked 
whether  locals  were  a  good  source  of  intelligence.  “Sometimes  they  were,  sometimes  they 
weren’t.  I  guess  it  really  depends.  I  mean  a  lot  of  them  just  looked  at  you  like  you’d  know  they 
knew. .  .if  they  were  looking  at  you  like  they  wanted  to  stay  out  of  it,  they’re  not  going  to  tell  you 
anything.  It’s  not  necessarily  that  they  don’t  want  to  tell  us,  it’s  that  they’re  scared  to  tell  us.” 
Another  Soldier  stated  “You’d  go  through  sectors  where  we’d  just  show  up  and  talk  to  the  sheik 
and  he’d  be  like  ‘There  aren’t  any  IEDs,’  like  there  really  aren’t,  or  he’d  tell  you  three  or  four 
spots,  kind  of  rat  the  Taliban  out.  And  then  you’d  go  to  other  sectors  and  it’s  not  as  reliable 
because  that  guy  is  less  supportive  of  American  and  the  Taliban  forces  are  stronger  in  his  area,  so 
he  can’t  support  more  even  if  he  wants  to.” 

Solutions, 

Less  experienced  Soldiers  did  not  provide  many  solutions  for  improving  threat  detection 
perfonnance.  Little  difference  existed  between  Soldiers  who  experienced  one  deployment  and 
those  who  had  been  on  multiple  deployments.  Across  both  groups,  Soldiers  stated  that  constant 
alertness  and  attention  to  detail  were  critical  to  effective  threat  detection.  One  Soldier  suggested 
that  foot  patrols  are  an  easy  way  to  reduce  complacency  while  increasing  mobility  and  avoiding 
attacks.  Two  Soldiers  made  comments  about  the  importance  of  making  connections  to  local 
populations.  The  first  Soldier  stated  “If  you’re  working  well  with  the  locals,  you’re  being 
generous  to  them,  and  you’re  working  hand-in-hand  with  them,  you’ll  have  no  problems  in  that 
area,  there’ll  be  no  issues  in  that  area.”  The  second  Soldier  stated  “Let’s  say  this  is  part  of  a 
sector  that  you’re  responsible  for,  I’d  probably  know  these  guys  standing  here  and  I  would  have 
a  reasonable  level  of  trust,  you  know  what  I  expect  whether  it  is  something  that  would  be  a  big 
threat  to  me  or  not.”  That  Soldier  discussed  how  they  might  counter  an  enemy  threat  “As  a 
mounted  force,  if  I  had  other  sectors  of  the  city  secured,  at  least  the  parallel  street,  to  put  forces 
at  the  viewpoint  of  the  camera,  then  kind  of  flank  around  to  the  rear  point  of  the  field  of  focus  as 
well.  That  would  be  ideal;  we’re  basically  taking  away  their  escape  route.”  Again,  these 
Soldiers  are  considering  strategies  to  reduce  threat  levels  rather  than  simply  spotting  cues. 

Summary  of  Research 

This  research  examined  how  less  experienced  and  more  experienced  Soldiers  performed 
experimental  exercises  related  to  threat  detection  and  responded  to  situations  that  contained 
potential  threats.  Some  of  the  key  findings  suggest  that  time,  threat  relevance,  and  expertise 
played  a  role  in  Soldiers’  performance.  Data  in  the  prioritized  threat  search  exercise 
demonstrated  that  novices  were  more  susceptible  to  time  pressure  and  the  number  of 
deployments  increased  accuracy  slightly  under  time  pressure.  Results  in  dynamic  threat 
detection  revealed  that  Soldiers  were  able  to  infer  threat-relevant  locations  and  appropriately 
focus  their  attention  to  those  areas.  Additionally,  all  Soldiers  had  a  tendency  to  identify  targets 
in  threat-relevant  locations.  Finally,  the  change  detection  exercise  indicated  that  Soldiers  were 
able  to  identify  changes  related  to  threats. 

An  analysis  of  the  interviews  revealed  differences  in  the  number  of  times  less 
experienced  and  more  experienced  Soldiers  discussed  strategies  for  threat  detection,  threat 
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detection  skills,  challenges  in  detecting  threats,  and  solutions.  Further  analysis  suggested  that 
there  were  differences  within  all  of  these  broad  categories,  ranging  from  topics  of  discussion  to 
the  content  of  these  discussions.  Novices  typically  responded  in  a  rule-based  fashion,  relying  on 
things  like  context-free  cues  or  threat,  whereas  experts  constructed  rich  narratives  and  scenarios 
that  allowed  them  to  describe  their  assessments  of  situations  in  ways  that  were  more  complex. 
Taken  together,  these  findings  suggest  ways  that  less  experienced  Soldiers  differ  from 
experienced  Soldiers.  We  can  use  these  differences  to  train  less  experienced  Soldiers  to  improve 
their  ability  to  identify  relevant  threats,  detect  relevant  changes,  and  develop  causal  reasoning 
skills  applicable  to  the  operational  environment. 

One  limitation  of  the  current  research  was  a  lack  of  clear  expert  and  novice  groups. 
Soldiers  ranged  from  zero  to  four  deployments.  Soldiers  fell  into  groups  of  relatively  even 
numbers  across  deployments  when  categorized  as  zero  deployments,  one  deployment,  and  2-4 
deployments.  Using  deployments  as  a  proxy  measure  of  experience  was  logical  for  this  research 
and  other  criteria  of  expertise,  such  as  years  in  the  military  and  rank  correlated  highly  with 
number  of  deployments.  Conducting  research  that  had  two  groups  with  clearly  delineated  levels 
of  experience  would  allow  for  a  clearer  analysis  of  experience  differences.  The  lack  of  distinct 
experience  levels  may  have  contributed  to  a  lack  of  statistical  significance  in  some  of  the 
analyses. 


Exemplar  Training  Development 


Design. 

Results  of  the  computer  exercises  demonstrated  the  utility  of  these  exercises  as  potential 
training  methods.  In  the  prioritized  threat  search  exercise,  time  pressure  negatively  affected  the 
perfonnance  of  Soldiers  with  zero  deployments,  but  had  little  or  even  a  positive  effect  on 
Soldiers  with  one  or  multiple  deployments.  Targets  in  threat-relevant  locations  were  located 
faster  and  with  greater  accuracy.  However,  no  statistically  significant  differences  were  found 
based  on  experience.  This  finding  indicated  that  Soldiers,  even  in  threat  search  exercises  using 
static  photos,  directed  their  search  to  relevant  threats  rather  than  to  random  locations  on  the 
photo  making  this  a  useful  stimulus  for  training.  Similarly,  in  the  change  detection  exercise 
participants  noticed  threat-relevant  changes  rather  than  threat-irrelevant  changes. 

The  interview  analysis  provided  insight  into  the  reasoning  of  less  and  more  experienced 
Soldiers.  As  expected,  Soldiers  with  less  experience  tended  to  make  context-free  procedurally- 
based  statements  whereas  experts  constructed  rich  narratives  to  describe  their  assessments  of 
situations.  This  analysis  revealed  differences  in  the  number  of  times  less  experienced  and  more 
experienced  Soldiers  discussed  strategies  for  threat  detection,  threat  detection  skills,  challenges 
in  detecting  threats,  and  solutions.  These  narratives  indicated  that  less  experienced  Soldiers  did 
not  have  the  mental  models  to  draw  on  to  interpret  the  situations  in  the  photos  even  if  they  could 
identify  relevant  threats.  Based  on  this  information,  the  training  incorporated  reasoning 
exercises  that  prompted  Soldiers  to  deliberate  over  threat  cues  and  use  information  about  the 
situation  and  environment  to  reason  about  likelihood,  threat  relevance,  and  severity.  This 
training  focused  on  helping  Soldiers  become  more  able  to  detect  and  interpret  threats  in  the 
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context  of  situations,  thus  allowing  them  to  broaden  their  experience  base  and  develop  applicable 
mental  models. 

From  the  research  data,  four  learning  objectives  were  identified.  These  learning 
objectives  map  onto  the  expected  behaviors  of  experienced  threat  detectors  identified  as  part  of 
the  threat  detection  loop  (Figure  1).  For  instance,  the  initial  exercises  focus  attention  on 
encoding  details  to  identify  and  classify  relevant  threats.  The  reasoning  exercises  increase  ability 
gather  evidence  to  determine  threat  importance  and  impact.  The  change  detection  exercises  were 
developed  to  increase  the  perception  of  changes  and  improve  the  ability  to  determine  the 
relevance  of  the  change  (Table  1 1). 

Table  1 1 

Relationship  between  learning  objectives  and  the  threat  detection  loop 


Learning  Objectives 

1 .  Demonstrate  an  improved 
ability  to  identify  relevant 
threats  in  a  variety  of 
situations. 

2.  Distinguish  between  threat¬ 
relevant  and  threat- 
irrelevant  cues  in  time- 
limited  situations. 


3.  Recognize  the  importance 
and  potential  impact  of 
each  detected  threat. 


4.  Identify  threat-relevant 
changes  in  the 
environment. 


Step  in  Threat  Detection 
_ Loop _ 

•  Monitoring,  vigilance, 
and  search  activities 


•  Identify/classify  threat 
or  non-threat 


•  Identify/classify  threat 
or  non-threat 

•  Gather  evidence  to  ID 
threat 


•  Change  or  anomaly 
detection 


Threat  Detection  Loop 
_ Behaviors _ 

a)  Encoding  details  for 
immediate  or  delayed  test 

b)  Ability  to  classify  threat  vs. 
non-threat  features 

i)  Ability  to  look  at  high- 
priority  threat  locations  based 
on  knowledge 

c)  Ability  to  adapt  decision 
selectivity  based  on 
information 

f)  Ability  to  balance  decision 
with  information  gathering 
h)  Ability  to  create  causal 
stories  about  threat  situations 

g)  Ability  to  detect  change  or 
anomaly 


To  determine  which  photos  to  use  in  the  training,  10  instructors  and  Soldiers  from  Fort 
Hood  reviewed  the  44  photos  and  selected  the  10  most  and  10  least  preferred  choices  to 
incorporate  into  training.  The  instructors  described  the  most  probable  threats  depicted  in  5  of 
their  top  10  photos  along  with  reasons  why  those  threats  would  be  the  most  difficult  to  detect. 
Soldiers  in  the  experiment  chose  some  of  the  same  photos  as  the  instructors.  Of  the  photos 
selected  by  Soldiers  in  the  experiment,  the  instructors  also  rated  many  in  the  top  10.  Focus  was 
given  to  photos  that  more  than  one  Soldier  commented  on  because  they  provided  threat  cue 
locations,  annotation,  and  interview  data  that  would  make  up  the  training  content.  The  photos 
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were  then  reviewed  for  training  applicability,  photo  quality,  and  the  ability  of  the  photo  to 
provide  training  content  that  met  the  learning  objectives.  The  interview  data  was  matched  to  the 
final  set  of  1 1  photos  and  then  photos  were  selected  that  would  apply  to  the  threat  search, 

o 

reasoning,  scenario,  and  change  detection  exercises  (Appendix  C). 

To  guide  the  development  process,  storyboards  provided  the  content  and  photos  that 
appear  on  each  screen  of  the  computer-based  training  (Appendix  D).  The  purpose  of  the 
storyboards  was  to  map  out  the  entire  course.  Military  subject  matter  experts  reviewed  the 
content  and  created  the  questions  for  the  Soldiers  to  answer  and  feedback  they  would  receive. 
Soldiers  also  were  given  feedback  about  their  threat  search  activities,  response  time,  and 
identification  of  threat  relevant  cues. 

The  exercises  in  the  training  take  a  crawl-walk-run  approach.  The  first  series  of  exercises 
involve  threat  searches.  Soldiers  first  view  photos  with  threat  cues  already  marked  and  click  on 
these  cues  to  read  about  what  the  threat  is,  why  the  threat  is  relevant,  and  common  enemy  tactics. 
Soldiers  then  view  photos  and  click  on  potential  threat  cues.  If  the  places  they  click  are  threats, 
annotations  pop  up,  again  providing  information  about  those  cues.  They  then  engage  in  a  timed 
threat  search  exercise  (Figure  10). 


MODULE  2:  THREAT  DETECTION 
Lesson  1 :  Identifying  Threats 


&  o 

TD  2-1-2  (2  of  7} 


IDENTIFYING  THREAT 

Exercise  #1 

Now  it's  your  turn. 

Click  on  locations  that 
could  conceal  a  threat. 
When  you  find  relevant 
threats,  a  red  area  will 
appear  with  information 
about  that  threat. 

When  you  are  done 
searching,  click  the 
" Finished "  button  below  to 
reveal  all  the  threats.  You 
must  click  on  at  least  three 
threat  locations  prior  to 
finishing.  You  have  lO  clicks 
maximum,  so  evaluate  the 
situation  carefully  before 
choosing  the  threat  area. 


Figure  10.  Screenshot  of  Threat  Search  Exercise. 


8 

Six  instructors  selected  three  of  these  photos  in  the  top  10  (each  of  these  photos  was  selected  once  in  the  bottom 
10)  and  four  selected  one  photo  in  the  top  10  (none  selected  it  in  the  bottom  10).  One  or  two  instructors  chose  five 
of  the  photos  in  the  top  10  (with  one  or  two  selecting  them  in  the  bottom  10).  Two  photos  did  not  appear  in  the  top 
or  bottom  10. 
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Following  the  threat  search,  Soldiers  view  photos  with  questions  that  require  them  to 
reason  about  the  situations  presented  and  evaluate  the  threats  in  context.  They  engage  in  critical 
thinking  about  the  potential  threats  and  read  several  possible  solutions  (Figure  1 1).  They  also 
read  comments  made  by  experienced  Soldiers  who  discussed  the  photos  during  the  interviews. 


MODULE  2:  THREAT  DETECTION 

Lesson  2:  Threat  Reasoning 
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2-2-8  (9  of  17) 


THREAT  REASONING 


Exercise  #2 


Create  a  story  about  this 
photo.  Consider  the 
reasoning  behind  each 
threat. 


1 .  What  might  be  the 
enemy's  motivation  for 
creating  such  a  threat? 


The  enemy  wants  to  inflict  casualties  on  friendly  forces.  The  enemy 
understands  that  to  win,  they  have  to  wear  down  the  U  S.  forces  and  harm 
U  S.  civilian  and  Soldier  morale  with  a  steady  stream  of  U  S.  casualties. 


Figure  11.  Screenshot  of  Reasoning  Exercise. 


Finally,  Soldiers  complete  the  threat  search  and  reasoning  exercises  by  reading  scenarios 
that  frame  the  context  of  the  photos  and  then  consider  that  context  to  identify  threats  and 
detennine  relevance  (Figure  12).  Soldiers  also  complete  two  change  detection  exercises  during 
the  training,  one  in  the  context  of  a  threat  search  and  one  in  the  context  of  causal  reasoning. 


Development. 

The  Instructional  Systems  Design  (ISD)  process  infonned  the  development  of  the 
computer-based  training  (CBT)  materials.  The  data  and  storyboards  contributed  to  the 
development  of  the  look,  content,  and  functionality  of  the  training  product.  A  Graphical  User 
Interface  (GUI)  was  created,  edited,  validated,  and  verified  and  then  a  Sharable  Content  Object 
Reference  Model  (SCORM)  compliant  version  of  the  CBT  course  was  developed.  When 
developing  a  SCORM  compliant  CBT  course,  the  major  sections  of  each  module,  such  as 
introductions,  lessons,  and  summaries  are  wrapped  into  their  own  Sharable  Content  Object 
(SCO).  The  SCORM  compliancy  allows  training  delivery  to  a  variety  of  Learning  Management 
Systems  (LMS)  that  the  military  currently  uses. 
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MODULE  2:  THREAT  DETECTION 

Lesson  3:  Threat  Relevance 


rtnri 


2-3-B.5  (8  of  17) 


SCENARIO  2: 
Urban  Terrain 


Click  on  top  3  most 
relevant  threats. 


Click  on  the  screen 
where  you  suspect  a 
threat  to  be  present.  If  a 
threat  is  present,  a  red 
area  will  appear. 


Figure  12.  Screenshot  of  Threat  Relevance  Exercise. 


While  in  development,  subject  matter  experts  (SME)  could  view  the  CBT  course  on  an 
internal  server  to  provide  comments,  request  changes,  and  check  the  course  for  functionality. 
The  evaluation  took  place  throughout  the  ISD  process  first  with  reviews  of  the  storyboards,  then 
through  initial  reviews  of  the  introduction  slides,  and  after  that,  a  formative  evaluation  process. 
After  the  evaluations  and  final  approval  of  the  course,  the  CBT  was  placed  on  an  external  server 
or  the  client-provided  LMS  and  CD/DVDs  were  created  for  distribution. 


Formative  evaluation. 


Three  instructors  at  Fort  Hood,  TX,  completed  two  evaluations  of  the  training  while  it 
was  under  development.  They  reviewed  an  early  draft  of  the  trainer  that  included  the 
introduction  and  the  pre-test  exercise.  Their  early  assessment  included  suggestions  for 
improving  the  narration,  providing  feedback  at  the  end  of  the  exercise,  and  collecting 
demographic  information.  These  reviewers  also  provided  feedback  about  the  CBT  format  and 
design.  These  reviewers  conducted  a  second  evaluation  of  a  more  advanced  version  of  the 
training  that  included  all  the  content  and  exercises,  but  was  missing  some  functionality.  The 
reviewers  answered  questions  specific  to  the  training  product  and  addressed  issues  of  training 
implementation.  The  following  questions  were  included. 

•  In  terms  of  being  able  to  interact  with  the  product,  are  there  certain  navigation  features 
that  need  improvement  (because  they  weren’t  easy  to  use/navigate)? 

•  Do  the  scenarios  and  photos  present  realistic  depictions  of  frequently  encountered 
situations  in  theater?  If  not,  do  you  have  recommendations  for  scenarios  and  photos  to 
include? 
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•  Who  would  be  the  best  audience  for  this  trainer:  new  Soldiers  prior  to  their  first 

deployment  or  as  a  refresher  for  Soldiers  who  are  deploying  after  one  or  more  previous 
deployments?  Why  did  you  choose  that  group? 

Overall,  the  responses  from  the  three  evaluators  were  very  positive.  Some  feedback 
addressed  functioning  issues  within  the  trainer  that  were  addressed  in  the  final  version.  The 
reviewers  appreciated  the  sense  of  urgency  that  time  restrictions  within  the  trainer  presented 
stating  that  although  it  is  not  the  same  as  being  in  the  actual  situation,  the  restriction  modeled 
time  stress  to  some  extent.  Other  responses  addressed  the  question  of  who  should  use  the  trainer. 
Suggestions  included  using  the  trainer  for  new  Soldiers  so  that  they  can  leam  important  skills 
before  deploying,  but  also  using  the  trainer  for  Soldiers  who  need  to  refresh  and  sharpen  skills 
before  redeployment.  Finally,  the  reviewers  suggested  that  the  trainer  in  combination  with  an 
instructor  would  provide  students  with  additional  insight  to  enhancing  threat  detection  skills. 

Suggestions  for  Methods  Refinements  and  Extensions 

Our  current  methodologies  used  static  imagery  to  both  identify  and  convey  threat  signals 
to  Soldiers  in  the  experiments  and  training  and  used  overt  annotation  and  search  behaviors  to 
collect  information  regarding  threats.  This  had  some  practical  advantages  in  that  Soldiers  could 
study  imagery  closely  without  time  pressure  and  they  could  make  annotations  in  detail. 

However,  new  technologies  may  augment  and  improve  both  data  collection  and  training  products 
in  three  important  ways:  1)  use  of  dynamic  video  imagery  for  improved  data  collection  and 
training,  2)  use  of  eye-tracking  hardware  and  other  means  to  identify  directly  the  locus  of 
attention  during  threat  detection  behavior,  and  3)  use  of  immersive  environments  to  improve 
training  realism. 

Dynamic  Video  Imagery.  The  use  of  static  imagery  allowed  us  to  use  high-resolution 
stimuli  which  contained  considerable  detail  that  is  often  lacking  from  the  video  footage  of  threat 
situations.  As  a  result,  our  methods  identified  threat  cues  that  were  primarily  static  such  as 
features  of  the  terrain  or  persons  in  the  scene  that  might  be  a  threat.  However,  our  interviews 
identified  a  number  of  threat  cues  that  are  dynamic  rather  than  static.  These  included  subtle 
interpersonal  cues  such  as  where  a  person  is  looking  or  how  he  or  she  is  walking,  ways  that  a 
vehicle  bounces  or  “drives  suspiciously,”  and  so  on.  Soldiers  can  only  identify  these  threat  cues 
in  dynamic  video  imagery  and  research  might  require  fundamentally  different  elicitation 
methods  to  uncover  this  type  of  information.  To  the  extent  that  it  is  possible  to  obtain 
appropriate  video  and  design  proper  data  collection  experiments,  training  can  be  further 
augmented  so  that  it  can  cover  both  static  and  dynamic  threat  cues. 

Attention-tracking  and  Eye-tracking  Hardware.  Several  of  the  research  exercises  used 
behavioral  cues  (mouse  clicks  or  identification  response  times)  to  help  identify  where  Soldiers 
were  directing  their  attention.  Software  or  hardware  systems  could  augment  these  exercises  to 
allow  closer  monitoring  of  visual  attention.  For  example,  simply  recording  the  location  of  the 
mouse  cursor  during  a  search  can  give  an  accurate  picture  of  where  a  Soldier  looks,  but  an  even 
more  promising  method  would  be  to  monitor  eye  movements  to  identify  exact  locations  of  search 
activities.  Eye -tracking  technologies  hold  particular  promise  if  combined  with  video  imagery 
because  research  can  begin  to  understand  more  detailed  visual  search  and  attention  strategies 


39 


during  threat  detection.  This  method  would  be  particularly  useful  to  identify  expert  threat  search 
strategies  in  complex  visual  environments  making  this  a  useful  technology  for  research  and  for 
training. 

For  example,  there  is  some  evidence  that  eye-tracking  technology  encourages  non¬ 
experts  to  adopt  visual  search  strategies  that  are  more  similar  to  those  of  experts.  Research  by 
Sadasivan,  Nalanagula,  Greenstein,  Gramopadhye,  and  Duchowski  (2004)  used  an  overlay  of  an 
expert’s  eye  movements  on  an  aircraft  that  required  inspection.  Participants  who  received  this 
type  of  training  demonstrated  greater  improvement  in  their  visual  search  perfonnance  compared 
to  those  who  did  not  receive  training.  This  feed-forward  training  allowed  participants  to  rely  on 
the  expert’s  search  strategy  early  in  the  visual  search  process,  thereby  adapting  to  a  more 
effective  search  technique.  Further  research  by  Nalanagula,  Greenstein,  and  Gramopadhye 
(2006)  used  three  types  of  displays  involving  eye-tracking  to  improve  perfonnance  when 
detecting  defects  in  circuit  boards.  These  three  displays  included  a  static  display  representing  the 
expert’s  eye  path  and  general  areas  of  interest,  a  dynamic  display  that  highlighted  the  expert’s 
areas  of  interest  in  the  order  perfonned  during  visual  search,  and  a  hybrid  display  combining 
both  static  and  dynamic  features.  Both  the  dynamic  and  hybrid  display  improved  the  number  of 
defects  participants  were  able  to  identify.  Eye-tracking  technologies  may  be  used  to  both 
identify  search  differences  between  novice  and  expert  threat  detectors  and  to  train  visual  search 
strategies  and  improve  skills.  With  the  advent  of  reasonably  cheap  eye-tracking  hardware, 
training  facilities  can  use  this  technology  as  a  method  to  improve  training  and  measure  skill 
level,  and  researchers  can  leverage  these  technologies  to  understand  threat  detection  strategies. 

Immersive  Training  Environments.  A  third  limitation  of  the  current  methodology  and 
training  is  that  it  focused  on  perceived  threats  rather  than  actual  threats.  The  imagery  used  in  the 
research  stimuli  and  training  did  not  contain  actual  attacks  or  IEDs.  While  the  purpose  of  the 
exercises  was  not  to  detect  actual  threats,  but  to  improve  the  cognitive  skills  associated  with 
threat  detection,  it  may  also  be  valuable  to  create  immersive  environments  that  simulate 
sequences  of  events  that  result  in  finding  and  disrupting  threats  and  dealing  with  the 
consequences  of  undetected  threats.  Training  threat  detection  in  field  environments  provides 
opportunity  for  Soldiers  to  move  through  situations  and  scan  360  degrees  from  a  first  person 
point  of  view.  The  current  prototype  offers  the  crawl  and  walk  stage  of  threat  detection  training 
while  immersive  training  environments  offers  the  run  stage  of  learning. 

At  least  three  types  of  immersive  environments  could  emerge  from  this  research:  live 
training,  video  exercises,  and  first-person  virtual  training.  Video-based  and  live  training  could 
be  similar  in  the  scenarios  they  present  and  the  skills  they  exercise.  Video-based  training 
provides  opportunity  to  present  select  scenarios  that  students  can  review  multiple  times  and  in 
conjunction  with  presentation  of  knowledge-based  training  and  knowledge  checks  (i.e.,  test 
questions,  and  critical  thinking  exercises).  Live  training  allows  students  to  engage  in  actual 
threat  detection  exercises  both  cognitively  and  physically  while  providing  a  safe  environment  to 
refine  skills  and  leam  from  mistakes.  The  data  collected  thus  far  can  contribute  to  the  creation  of 
training  scenarios  and  feedback  guides  for  use  in  existing  live  training  such  as  current  Lanes 
Training  Exercises  (LTX),  Field  Training  Exercises  (FTX),  or  similar  skill  training  events. 
Video-based  training  can  leverage  live  scenario  materials  as  a  basis  for  selecting  dynamic  video 
imagery  that  would  drive  off-line  training  via  computer  or  classroom  settings.  Finally, 
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developing  virtual  environments  would  allow  for  dynamic  generation  of  threats,  allowing 
situations  to  replay  so  students  can  identify  threat  cues  they  missed  and  engage  in  after  action 
reviews  etc.  Virtual  environments  also  allow  for  automated  feedback  and  dynamic  generation  of 
content. 


Conclusion 

The  series  of  research  phases  conducted  to  understand  Soldier  threat  detection  provides  a 
better  understanding  of  how  Soldiers  manage  attention,  process  information,  and  reason  about 
events  they  encounter  in  the  OE.  The  first  three  phases  of  this  research  provided  insight  into  the 
key  threats  Soldiers  consider  important  and  clarified  the  difficulty  Soldiers  have  in  distinguishing 
ambiguous  cues  from  certain  cues  when  searching  for  threats.  Data  from  interviews  with 
experienced  Soldiers  enabled  the  creation  of  a  threat  detection  loop  model  and  identification  of 
the  behaviors  necessary  for  proficient  threat  detection.  The  outcome  of  the  computerized 
exercises  was  stimuli  for  the  final  research  that  included  key  threat  locations  and  cues  in  photos 
and  critical  situational  factors  that  influence  threat  relevance.  The  aim  of  the  final  research  phase 
was  to  compare  the  threat  detection  performance  of  less  and  more  experienced  Soldiers.  This 
research  demonstrated  the  cognitive  skills  Soldiers  employ  when  searching  for  threat -relevant 
cues  and  changes  in  situations  with  and  without  time  pressure.  Interviews  also  provided  data  that 
mapped  onto  the  threat  detection  loop  and  contextual  details  that  informed  our  knowledge  of  the 
threat  detection  process. 

While  all  of  this  research  provided  contextual  detail  about  how  Soldiers  think  about 
threat  cues,  threat  detection  activities,  attention  management,  and  search  priorities,  the  final 
research  phase  contributed  empirical  evidence  of  Soldier  threat  detection  abilities.  This  research 
showed  that  time  pressure  influences  Soldier  ability  to  detect  threats  with  fewer  threats  being 
detected  when  under  time  pressure.  These  results  provided  some  indication  that  experience 
reduces  the  effects  of  time  pressure,  but  this  effect  was  small  and  thus  more  research  is  required. 
Regardless  of  experience,  Soldiers  focused  their  attention  toward  detecting  relevant  threats 
compared  to  irrelevant  threats.  They  were  more  accurate  and  responded  faster  to  relevant  threats 
indicating  that  targeted  searching  in  relevant  locations  is  a  component  of  threat  search  exercises. 
While  experience  did  not  mediate  this  effect,  the  interview  data  indicated  that  Soldiers  with 
deployment  experience  had  a  deeper  understanding  of  why  threats  were  relevant  and  under  what 
circumstances  threat  relevance  would  increase  or  decrease.  They  were  able  to  construct  stories 
around  the  threats,  reason  about  various  action  choices,  and  hypothesize  about  potential 
outcomes. 

The  change  detection  exercise  was  one  of  the  more  informative  concerning  Soldier  ability 
to  determine  threat  relevance.  Although  results  revealed  a  high  false  alarm  rate  when  trying  to 
find  changes  in  the  photos  presented,  when  Soldiers  accurately  identified  changes  the  majority 
were  threat-relevant  changes.  This  finding  adds  to  the  existing  change  detection  literature  by 
demonstrating  Soldier  ability  to  notice  changes  at  rates  greater  than  chance.  Previous  research 
would  suggest  that  the  Soldiers  should  have  a  difficult  time  detecting  changes.  Change  blindness 
is  a  phenomenon  where  individuals  are  incapable  of  detecting  salient  changes  in  their  visual 
environment  (O’Regan,  1992;  Rensink,  O’Regan,  &  Clark,  1997;  Simons  &  Rensink,  2005). 
Previous  research  has  demonstrated  that  observers  fail  to  notice  large  changes  made  during  eye 
movements,  across  a  flicker  paradigm,  during  repeated  viewings  of  the  visual  scene,  and  when 
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conversational  partners  changed  during  a  real-life  interaction.  Our  research  demonstrates  that  at 
an  applied  level  Soldiers  demonstrate  some  ability  to  detect  change  above  chance  levels. 

Researchers  have  identified  some  situations  that  mitigate  change  blindness. 

Hollingsworth  (2004)  and  Brady,  Konkle,  Oliva,  and  Alvarez  (2009)  suggest  that  intentional 
encoding  enables  individuals  to  attend  to  changes.  However,  in  the  current  research,  Soldiers 
received  no  explicit  instructions  to  suggest  they  should  study  the  photograph  for  subsequent 
testing.  The  test  to  detect  changes  was  a  surprise  for  these  participants  and  their  performance 
suggests  an  enhanced  ability  to  detect  changes.  Further,  it  should  have  been  difficult  for  Soldiers 
to  detect  changes  because  the  subsequent  testing  after  the  target  picture  provided  significant 
interference.  Makovski,  Shim,  and  Jiang  (2006)  tested  change  detection  ability  in  natural  scenes 
after  presenting  blank,  visual,  or  auditory  delays  and  requiring  participants  either  to  view 
passively  the  delay  screens  or  attend  to  the  information  in  the  delay.  Participants  who  attended 
to  the  delay  infonnation  had  a  significant  reduction  in  their  change  detection  ability.  Because 
information  presented  between  the  target  photograph  and  the  changed  photograph  required 
Soldiers  to  attend  to  a  significant  amount  of  information,  it  was  expected  that  they  would  have 
difficulty  detecting  changes  during  the  second  presentation  of  the  photos.  Further  research 
should  attempt  to  provide  an  explanation  for  this  finding. 

Experts  have  been  shown  to  detect  changes  more  often  than  novices  do  when  the  change 
is  related  to  their  domain  of  expertise  (Jones,  Jones,  Smith,  &  Copley  2003;  Wemer  &  Thies, 
2000).  In  the  current  research,  we  found  no  differences  in  change  detection  perfonnance  based 
on  experience.  The  data  did  not  provide  insights  into  why  no  differences  existed.  It  could  be 
that  general  training  related  to  detecting  threats  made  Soldiers  aware  of  the  potential  for  change 
in  their  environments  and  thus  they  were  attuned  to  looking  for  changes.  It  could  also  be  that 
threat-relevant  cues  are  emotional  stimuli  for  Soldiers  and  those  cues  captured  their  attention 
(Mayer,  Muris,  Vogel,  Nojoredjo,  &  Merchelbach,  2006;  Peira,  Golkar,  Larsson,  &  Wiens,  2010; 
Vuilleumier,  2005)  thus  making  the  presence  or  absence  of  such  cues  salient.  It  may  also  be  that 
the  distinction  between  the  experienced  and  novice  groups  was  not  clear  in  the  current  research. 
Future  research  should  attempt  to  identify  the  cause  of  these  findings. 

Results  of  this  research  provide  an  increased  understanding  of  threat  detection  and 
directions  for  future  research  and  training  exemplar  development.  The  current  training  uses 
static  images  and  exercises  developed  during  research  that  also  used  static  images  as  stimuli  to 
inform  our  knowledge  of  Soldier  threat  detection.  Future  research  and  training  should  explore 
the  use  of  video  images  and  eye  tracking  to  gather  data  to  infer  perception  and  attention 
processes  and  to  train  threat  detection  in  complex  and  dynamic  environments.  Soldiers  of 
various  skill  levels  tend  to  focus  on  relevant  threats;  however,  experience  adds  context  and 
reasoning  to  the  threat  identification  process.  Future  training  should  continue  to  highlight  the 
causal  reasoning  and  critical  thinking  components  of  threat  detection.  This  research  indicated 
that  Soldiers  have  ability  to  notice  changes,  but  this  ability  was  limited  relative  to  the  high  false 
alann  rate.  Although  change  detection  accuracy  was  not  random,  it  may  be  possible  to  improve 
this  existing  skill  with  targeted  training. 
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Appendix  A 


Demographic  Questions 
Fort  Hood  -  13-15  December  2010 


1.  Time  in  service  (years) 

2.  Current  rank 

3.  Time  in  current  rank  (months) 

4.  Current  MOS 

5.  Age 

6.  Have  you  ever  deployed?  Yes  No 

7.  How  many  times  have  you  deployed? 

8.  Location  of  most  recent  deployment  (city  or  cities  and  country) 

9.  MOS  while  on  most  recent  deployment 

10.  How  often  did  you  go  outside  the  wire  on  your  most  recent  deployment? 

Never  Less  than  once  a  month  Once  a  month  More  than  once  a  month 
Once  a  week  More  than  once  a  week  Everyday 

1 1 .  Describe  some  of  your  duties  during  your  most  recent  deployment: 


A-l 


Appendix  B 


Overlay  indicating  locations  Soldiers  clicked  on  to  identify  potential  threats 


B-l 


Appendix  C 


Photos  selected  for  the  threat  detection  exercises 


Threat  Search 


Reasoning 


Pre  /Post  Test 


C-l 


Change  Detection 


C-2 


Appendix  D 


Storyboard  example  used  in  training  exemplar  development 


Course: 

Threat  Detection  Skills  Trainer 

Module: 

Threat  Detection 

TD 

Lesson: 

Identifying  Threats 

2 

TD2-1-2 

Segment: 

1 

Page  Title: 

Identifying  Threat  Exercise  #1 

9 

Child  Page: 

Objective: 

1)  Identify  relevant  threats  in  a  variety  of  military  patrol  contexts 

On-Screen  Text: 


Now  it’s  your  turn.  Click  on  locations  that  could  conceal  a  threat.  When  you  find  relevant  threats,  a  red 
dot  will  appear  with  information  about  that  threat. 

When  you  are  done  searching,  click  the  “Finished”  button  to  reveal  all  the  threats.  You  must  click  on  at 
least  three  threat  locations  prior  to  finishing.  You  have  10  clicks  maximum,  so  evaluate  the  situation 
carefully  before  choosing 


Narration/Closed  Captioning:  Narrator 

Can  you  find  the  relevant  threats  in  this  photo? 


Graphics:  (P  -  photo;  G  -  graphic;  F  -  flash  animation;  T  -  table/chart/graph;  V  -  video) 
Use  photo:  Clean  34  from  treat-hires  folder. 

Use  information  from  Threat  Photos-Annotations:  Slide  06  for  threat  areas  descriptions. 
(Place  the  red  dot  in  the  middle  of  each  threat  area.) 


Audio: 


Knowledge  Check:  Remedial  Screen:  Page  ID 


Correct  Feedback: 

Provide  the  number  of  clicks  the  student  made  and  the  number  of  threats  found 

1st  try  incorrect: 

2nd  try  incorrect: 

Explanatory  Information: 


When  Soldiers  click  on  threat  spots,  show  a  red  dot  in  that  spot.  The  spot  should  remain  visible. 
Make  each  red  dot  a  pop-up  of  threat’s  description,  with  ability  to  click  and  close 
After  student  finishes,  show  all  red  dots  and  pop-ups.  Do  not  allow  them  to  click  finish  until  they  have 
made  at  least  three  clicks 


Branching:  Back: 


Next: 


D-l 


